Introduction
This tutorial presents an end-to-end collaborative DataStage development workflow designed to transform traditional ETL development into a modern, version-controlled, and team-based process. It combines Git version control, DataStage ETL capabilities, and collaborative development practices to enable multiple developers to work seamlessly and continuously on complex data integration projects.
By operationalising version control through Git integration, organizations can enhance code quality, enable parallel development, maintain complete audit trails, and confidently move from development to production deployment with full traceability.
Use Case
Healthcare organizations manage vast volumes of patient data including electronic health records (EHRs), clinical transactions, and patient demographics across multiple systems. Without version control, teams struggle to collaborate on IBM DataStage ETL projects, leading to overwritten changes, no audit history, and unreliable deployments.
A Git-integrated DataStage solution enables teams to work on patient clinical analytics pipelines collaboratively creating branches, making independent changes, reviewing each other's work, and merging validated code to production with full traceability.
For this use case, enterprises can leverage IBM DataStage with Git Integration for version-controlled ETL development, GitHub for repository management, and pull request workflows for code review and quality assurance.
Problem Statement
In large healthcare organizations, data engineers from different departments — such as Clinical Operations, Patient Demographics, and Billing & Transactions — independently develop and modify shared DataStage ETL pipelines. When multiple developers modify the same flows simultaneously without version control, changes conflict and overwrite each other, causing pipeline failures and data loss. Without branching, teams cannot work in parallel, and without audit trails, there is no traceable history of changes — a serious compliance risk in regulated healthcare environments. Manual coordination across departments further leads to integration failures, and reverting faulty transformations in production remains slow and error-prone.
Solution
IBM DataStage with Git Integration resolves these challenges by bringing version control directly into the ETL development lifecycle. Each department works on an isolated feature branch, eliminating conflicts and enabling parallel development. Every change is tracked with author, timestamp, and commit message — meeting healthcare compliance requirements. Pull request workflows enforce peer review before any change reaches the main branch, ensuring data quality and pipeline integrity. Developers can clone and sync a colleague's branch directly within DataStage, and any faulty change can be instantly rolled back to a stable state if a production issue is detected.
Architectural Diagram
Persona Setup
We recommend using the following roles for this tutorial:
Steps to create user groups:
The Administrator creates custom platform roles.
- Login with admin credentials and navigate to Access Control from the Software Hub Perspective.
- Go to User group tab, click New User Group, enter group name "Data Scientist", add users, assign out-of-the-box roles Data Scientist Role, Manage Glossary, Manage Governance Categories, Manage Catalogs, then save the group.
Follow the same step and create rest of the groups.
For the capabilities of the user — refer to IBM documentation: https://www.ibm.com/docs/en/software-hub/5.1.x?topic=a-managing-users
Each user is then assigned specific roles that define their capabilities and permissions within the DataStage environment.
- Data Engineer (admin): Create and configure the Git repository and manage user access.
- Data Scientist 1 (user1): Creates the initial DataStage project, develops ETL flows, and manages the main development workflow.
- Data Scientist 2 (user2): Collaborates on the project, enhances existing flows, and contributes improvements.
Step 1: Create Git Repository and Add Collaborators
- Log in to your Git platform (GitHub/GitLab/Bitbucket) as administrator.
- Click on "New Repository".
- Enter repository name: Healthcare-Patient-Analytics-ETL.
- Add description: "DataStage ETL pipeline for patient clinical analytics".
- Set visibility: Private (recommended for enterprise projects).
- Initialize with README (optional) and create Repository.
Add Collaborators:
- Navigate to Repository Settings → Collaborators or Members.
- Add Data Scientist 1 (user1) with Admin or Write access.
- Add Data Scientist 2 (user2) with Write access.
- Send invitations to both users. Create a feature branch patient-clinical-analytics-pipeline in repository.
Fig 1: Git Repository and Collaborator Setup
Step 2: Data Scientist 1 Creates Git-Integrated DataStage Project (PatientAnalytics_Dev_DS1)
- Log in as Data Scientist 1 (user1).
- From IBM Cloud Pak for Data navigation menu, select Projects > View all projects.
- Click "New project."
- Enter project name: PatientAnalytics_Dev_DS1.
- Add description: "Patient clinical analytics ETL development — Data Scientist 1."
- Click "Create."
Configure Git Integration:
- In the project, navigate to Settings → Git Integration.
- Click "Enable Git Integration."
- Enter Git repository URL: https://github.com/healthcare/PatientAnalytics_DataStage.git.
- Select authentication method: Personal Access Token.
- Enter credentials.
- Set default branch: patient-clinical-analytics-pipeline.
Fig 2: Git Integration in Project
Step 3: Create DataStage Flow
- Log in as Data Scientist 1 (user1).
- In PatientAnalytics_Dev_DS1 project, navigate to Assets tab.
- Click "New asset" → Select "Transform and integrate data."
- Name: seq_PatientClinicalAnalytics_Master.
- Add description: "Master sequence orchestrating patient clinical analytics pipeline" and create the flow.
Add Source Stage — PATIENT_CLINICAL_TRANSACTIONS:
- Stage type: Connector stage.
- Connection: db2oncloud.
- Table: PATIENT_CLINICAL_TRANSACTIONS.
Add Source Stage — PATIENT_DEMOGRAPHICS:
- Stage type: Connector stage.
- Connection: db2oncloud.
- Table: PATIENT_DEMOGRAPHICS.
Add Join Stage — Join_1:
- Stage type: Join stage.
- Purpose: Join patient clinical transactions with patient demographics on a common key.
- Join type: Inner Join.
- Join key: PATIENT_ID.
- Left input link: PATIENT_CLINICAL_TRANSACTIONS.
- Right input link: PATIENT_DEMOGRAPHICS.
Add Target Stage — IBM_Db2_on_Cloud:
- Stage type: Connector stage.
- Connection: db2oncloud.
- Table: PATIENT_CLINICAL_ANALYTICS_OUTPUT (target table).
- Write mode: Replace (or Append based on requirement).