IBM Cloud Pak for Data

IBM Cloud Pak for Data

Come for answers. Stay for best practices. All we’re missing is you.

 View Only

Enterprise Version Control for DataStage: Building DevOps-Ready Git Workflows

By JETLIN Yeldo posted 09/30/26 06:36 AM

  

Introduction

This tutorial presents an end-to-end collaborative DataStage development workflow designed to transform traditional ETL development into a modern, version-controlled, and team-based process. It combines Git version control, DataStage ETL capabilities, and collaborative development practices to enable multiple developers to work seamlessly and continuously on complex data integration projects.

By operationalising version control through Git integration, organizations can enhance code quality, enable parallel development, maintain complete audit trails, and confidently move from development to production deployment with full traceability.

Use Case

Healthcare organizations manage vast volumes of patient data including electronic health records (EHRs), clinical transactions, and patient demographics across multiple systems. Without version control, teams struggle to collaborate on IBM DataStage ETL projects, leading to overwritten changes, no audit history, and unreliable deployments.

A Git-integrated DataStage solution enables teams to work on patient clinical analytics pipelines collaboratively creating branches, making independent changes, reviewing each other's work, and merging validated code to production with full traceability.

For this use case, enterprises can leverage IBM DataStage with Git Integration for version-controlled ETL development, GitHub for repository management, and pull request workflows for code review and quality assurance.

Problem Statement

In large healthcare organizations, data engineers from different departments — such as Clinical Operations, Patient Demographics, and Billing & Transactions — independently develop and modify shared DataStage ETL pipelines. When multiple developers modify the same flows simultaneously without version control, changes conflict and overwrite each other, causing pipeline failures and data loss. Without branching, teams cannot work in parallel, and without audit trails, there is no traceable history of changes — a serious compliance risk in regulated healthcare environments. Manual coordination across departments further leads to integration failures, and reverting faulty transformations in production remains slow and error-prone.

Solution

IBM DataStage with Git Integration resolves these challenges by bringing version control directly into the ETL development lifecycle. Each department works on an isolated feature branch, eliminating conflicts and enabling parallel development. Every change is tracked with author, timestamp, and commit message — meeting healthcare compliance requirements. Pull request workflows enforce peer review before any change reaches the main branch, ensuring data quality and pipeline integrity. Developers can clone and sync a colleague's branch directly within DataStage, and any faulty change can be instantly rolled back to a stable state if a production issue is detected.

Architectural Diagram

Persona Setup

We recommend using the following roles for this tutorial:

Steps to create user groups:


The Administrator creates custom platform roles.

  • Login with admin credentials and navigate to Access Control from the Software Hub Perspective.
  • Go to User group tab, click New User Group, enter group name "Data Scientist", add users, assign out-of-the-box roles Data Scientist Role, Manage Glossary, Manage Governance Categories, Manage Catalogs, then save the group.

Follow the same step and create rest of the groups.

For the capabilities of the user — refer to IBM documentation: https://www.ibm.com/docs/en/software-hub/5.1.x?topic=a-managing-users

Each user is then assigned specific roles that define their capabilities and permissions within the DataStage environment.

  • Data Engineer (admin): Create and configure the Git repository and manage user access.
  • Data Scientist 1 (user1): Creates the initial DataStage project, develops ETL flows, and manages the main development workflow.
  • Data Scientist 2 (user2): Collaborates on the project, enhances existing flows, and contributes improvements.

Step 1: Create Git Repository and Add Collaborators

  • Log in to your Git platform (GitHub/GitLab/Bitbucket) as administrator.
  • Click on "New Repository".
  • Enter repository name: Healthcare-Patient-Analytics-ETL.
  • Add description: "DataStage ETL pipeline for patient clinical analytics".
  • Set visibility: Private (recommended for enterprise projects).
  • Initialize with README (optional) and create Repository.

Add Collaborators:

  • Navigate to Repository Settings → Collaborators or Members.
  • Add Data Scientist 1 (user1) with Admin or Write access.
  • Add Data Scientist 2 (user2) with Write access.
  • Send invitations to both users. Create a feature branch patient-clinical-analytics-pipeline in repository.
Fig 1: Git Repository and Collaborator Setup

Step 2: Data Scientist 1 Creates Git-Integrated DataStage Project (PatientAnalytics_Dev_DS1)

  • Log in as Data Scientist 1 (user1).
  • From IBM Cloud Pak for Data navigation menu, select Projects > View all projects.
  • Click "New project."
  • Enter project name: PatientAnalytics_Dev_DS1.
  • Add description: "Patient clinical analytics ETL development — Data Scientist 1."
  • Click "Create."

Configure Git Integration:

  • In the project, navigate to Settings → Git Integration.
  • Click "Enable Git Integration."
  • Enter Git repository URL: https://github.com/healthcare/PatientAnalytics_DataStage.git.
  • Select authentication method: Personal Access Token.
  • Enter credentials.
  • Set default branch: patient-clinical-analytics-pipeline.
Fig 2: Git Integration in Project

Step 3: Create DataStage Flow

  • Log in as Data Scientist 1 (user1).
  • In PatientAnalytics_Dev_DS1 project, navigate to Assets tab.
  • Click "New asset" → Select "Transform and integrate data."
  • Name: seq_PatientClinicalAnalytics_Master.
  • Add description: "Master sequence orchestrating patient clinical analytics pipeline" and create the flow.

Add Source Stage — PATIENT_CLINICAL_TRANSACTIONS:

  • Stage type: Connector stage.
  • Connection: db2oncloud.
  • Table: PATIENT_CLINICAL_TRANSACTIONS.

Add Source Stage — PATIENT_DEMOGRAPHICS:

  • Stage type: Connector stage.
  • Connection: db2oncloud.
  • Table: PATIENT_DEMOGRAPHICS.

Add Join Stage — Join_1:

  • Stage type: Join stage.
  • Purpose: Join patient clinical transactions with patient demographics on a common key.
  • Join type: Inner Join.
  • Join key: PATIENT_ID.
  • Left input link: PATIENT_CLINICAL_TRANSACTIONS.
  • Right input link: PATIENT_DEMOGRAPHICS.

Add Target Stage — IBM_Db2_on_Cloud:

  • Stage type: Connector stage.
  • Connection: db2oncloud.
  • Table: PATIENT_CLINICAL_ANALYTICS_OUTPUT (target table).
  • Write mode: Replace (or Append based on requirement).

Step 4: Compile DataStage Jobs

  • Log in as Data Scientist 1 (user1).
  • In DataStage Designer, select all jobs in the project.
  • Right-click → Select "Compile."
  • Wait for compilation to complete.
  • Review compilation results.
  • Check for any errors or warnings.
  • Resolve any issues if found, recompile if necessary.

Verify Compilation Success:

  • Check that all jobs show "Compiled successfully" status.
  • Review compilation log for any warnings.
  • Ensure all dependencies are resolved.

Step 5: Run and Test DataStage Flow

  • Log in as Data Scientist 1 (user1).
  • Open seq_PatientClinicalAnalytics_Master sequence.
  • Click "Run" button.

Monitor Execution:

  • Watch job progress in real-time.
  • Check each stage completion status.
  • Monitor row counts and processing statistics.
  • Review any warnings or informational messages.

Validate Results:

  • Check output datasets for data quality.
  • Verify row counts match expectations.
  • Validate transformation logic results.
  • Review logs for any issues.
Fig 4: Executing Datastage Flow

Step 6: Commit Changes to Git Repository

  • Log in as Data Scientist 1 (user1).
  • In DataStage project, navigate to Manage DataStage → Choose Git Integration → Sync.
  • Choose 'Commit changes'.

Stage Changes:

Review the list of new and modified assets.

Fig 5: Reviewing Modified Assets

Commit with Detailed Message:

  • Click "Commit."
  • Verify Commit is successful.
Fig 6: Committing with Detailed Message

Step 7: Data Scientist 2 Creates DataStage Project (HealthcareETL_Dev_DS2)

  • Log in as Data Scientist 2 (user2).
  • From IBM Cloud Pak for Data navigation menu, select Projects > View all projects.
  • Click "New project."
  • Enter project name: HealthcareETL_Dev_DS2.
  • Add description: "Healthcare ETL development — Data Scientist 2."
  • Click "Create."

Configure Git Integration:

  • In the project, navigate to Settings → Git Integration.
  • Click "Enable Git Integration."
  • Enter same Git repository URL: https://github.com/healthcare/PatientAnalytics_DataStage.git.
  • Select authentication method: Personal Access Token.
  • Enter Data Scientist 2's credentials.
  • Set default branch: patient-clinical-analytics-pipeline.
Fig 7: Data Scientist 2 Project and Git Integration

Step 8: Data Scientist 2 Clones Data Scientist 1's Feature Branch

  • Log in as Data Scientist 2 (user2).
  • Open HealthcareETL_Dev_DS2 project.
  • In DataStage project, navigate to Manage DataStage → Choose Git Integration → Sync.
  • Choose 'Clone from GitHub.'
Fig 8: Cloning from GitHub
Fig 9: Repository Clone

Verify Assets:

  • Navigate to Assets tab.
  • Confirm all of Data Scientist 1's DataStage assets are now visible.

Step 9: Data Scientist 2 Modifies and Enhances DataStage Flow

  • Log in as Data Scientist 2 (user2).
  • Open the DataStage assets in HealthcareETL_Dev_DS2 project.

Enhancements Made:

  • Data Scientist 2 adds enhancements to the existing DataStage flow.
  • Additional Transformations: Added one filter stage for one of the columns with a WHERE clause.

Fig 10: Enhanced DataStage Flow with Filter Stage

Step 10: Data Scientist 2 Compiles and Runs Updated Flow

  • In DataStage Designer, select all modified jobs.
  • Right-click → Select "Compile."
  • Wait for compilation to complete.
  • Verify all jobs compile successfully.

Run Complete Flow:

  • Open seq_PatientClinicalAnalytics_Master.
  • Click "Run."
  • Monitor execution of all jobs including new enhancements.
Fig 11: Executing Enhanced Datastage Flow

Step 11: Data Scientist 2 Commits Enhancements

  • In DataStage project, navigate to Manage DataStage → Choose Git Integration → Sync.
  • Choose 'Commit changes.'
  • Review all modified assets.

Stage Changes:

  • Select all modified jobs and assets.
  • Click "Commit."

Fig 12: Staging Enhanced Changes

Commit with Message:

  • Click "Commit."
  • Enter commit message.
  • Verify the commit is successful.
Fig 13: Committing Enhancements
Fig 14: Successful Commit Confirmation

Step 12: Data Scientist 1 Pulls Data Scientist 2's Updates and Reviews Changes

  • Log in as Data Scientist 1 (user1).
  • Open PatientAnalytics_Dev_DS1 project.
  • In DataStage project, navigate to Manage DataStage → Choose Git Integration → Sync.
  • Choose 'Clone from GitHub.'

Review Changes:

  • Navigate to Assets tab.
  • Review Data Scientist 2's new and modified DataStage flow.

Fig 15: Reviewing Data Scientist 2's Updates

Step 13: Data Scientist 1 Tests Complete Integrated Flow

  • Log in as Data Scientist 1 (user1).
  • Open seq_PatientClinicalAnalytics_Master sequence.
  • Verify the DataStage flow with updated Filter Enhancement.
  • Compile and Run the DataStage Flow.
Fig 16: Testing Complete Integrated Flow

Run End-to-End Test:

  • Click "Run."
  • Monitor complete pipeline execution.
  • Verify all jobs execute successfully.

Validate Integration:

  • Verify Data Scientist 1's original logic still works correctly.
  • Confirm Data Scientist 2's enhancements integrate properly.
  • Check data consistency across all transformations.
  • Validate output datasets.

Step 14: Create Pull Request to Main Branch

  • Log in as Data Scientist 1 (user1).
  • Navigate to Git repository (GitHub/GitLab/Bitbucket) in web browser.
  • Go to repository: Healthcare-Patient-Analytics-ETL.

Create Pull Request:

  • Click "New Pull Request" or "Create Merge Request."
  • Source branch: feature/patient-clinical-analytics.
  • Target branch: main.
  • Enter PR title: "Patient Clinical Analytics ETL Pipeline — Production Ready."

Add Detailed Description:

  • Complete ETL pipeline for patient clinical analytics.

Features Implemented:

  • End-to-end data ingestion from patient demographics and clinical transactions.
  • Comprehensive data quality validations and cleansing.
  • Patient segmentation with risk scoring logic.
  • Performance optimizations (40% improvement in processing time).
  • Parallel processing for scalability.
  • Enhanced error handling and logging.
  • Complete documentation and metadata.

Collaborators: @DataScientist1 @DataScientist2

Testing Completed:

  • ✓ Unit testing of all individual jobs.
  • ✓ Integration testing of complete pipeline.
  • ✓ Performance benchmarking.

Add Reviewers:

  • Select team lead as reviewer.

Add Labels:

  • Enhancement.
  • Click "Create Pull Request."
Fig 17: Creating Pull Request

Step 15: Code Review and Approval Process

Team Lead Reviews Pull Request:

  • Log in as Team Lead.
  • Navigate to Pull Request.
  • Review all changed files and DataStage jobs.
  • Check code quality and adherence to standards.
  • Validate transformation logic.
  • Review documentation completeness.

Add Review Comments:

  • Click on specific jobs to add inline comments.
  • Request clarifications if needed.
  • Suggest improvements (optional).
  • Approve changes.

Senior Data Engineer Review:

  • Review performance optimizations.
  • Validate data quality rules.
  • Check error handling implementation.
  • Verify scalability considerations.
  • Add approval.

Final Approval:

  • All reviewers approve the pull request.
  • Status changes to "Approved."
  • Ready for merge.
Fig 19: Final Approval Status

Conclusion

In this tutorial, we built an end-to-end collaborative DataStage development workflow using Git integration, enabling Data Scientist 1 and Data Scientist 2 to work seamlessly on a patient clinical analytics ETL pipeline. By leveraging version control, branching strategies, code review processes, and controlled deployments, the solution enables consistent, high-quality data integration development with complete traceability.

Key Benefits:

Using this Git-integrated approach, development teams can:

  • Work in parallel without conflicts.
  • Maintain complete version history and audit trails.
  • Implement code review for quality assurance.
  • Deploy with confidence using tested and validated code.
  • Roll back changes quickly if issues arise.
  • Scale development across distributed teams.

The patient clinical analytics pipeline processes healthcare data through multiple stages — data extraction, quality validation, transformation, patient segmentation, and shaped dataset creation — all while maintaining enterprise-grade governance and collaboration practices.

This approach provides a scalable and governed foundation for operationalizing DataStage development across healthcare and other enterprise use cases, transforming traditional ETL development into a modern, collaborative, and version-controlled process.

0 comments
39 views

Permalink