Back to Blog List

Migrating to IBM Dependency Based Build 2.0 (Part 2)

Like

PART 2 - Prepare CI/CD pipelines for DBB 2.0

This blog entry is Part 2 of a 3-part series of posts discussing the migration process from DBB 1.1 to DBB 2.0 and its implications. The first post, Part 1 - Migrate existing datastore managed in Derby to Db2z, covered the database migration process from Derby to Db2. This migration step is optional if you are already using a supported database for DBB 2.0 (that is, either Db2 LUW or Db2 z/OS).

Note: This blog series was originally published in early 2023, and only applies to migrations from DBB 1.x to DBB 2.0, and not migrations from DBB 1.x directly to DBB 2.0.1 or above. In June 2023, DBB 2.0.1 was released and introduced database schema changes that do not allow a DBB 1.x and DBB 2.0.1 toolkit to access the same database instance. As a reminder, DBB 1.x will be going out of support at the end of April 2024. If you need additional advice on migrating from DBB 1.x, please open a support ticket with IBM Support.

Our team has planned and executed the migration of our DevOps demo environment to DBB 2.0 with a main goal of minimizing the impact of the migration to the GitLab pipelines and applications running in the demo environment - just as a production environment would require.

The official DBB documentation includes an overview of the different activities when upgrading to DBB 2.0. In this post, we will go through the update of the customized build framework and the analysis that we performed to understand how DBB is used within our GitLab pipelines.

We grouped these activities into two main parts:

These steps would typically involve system programmer and security roles for setting up DBB 2.0, as well as build engineer and DevOps engineer roles for updating the build processes and pipeline definitions.

Update our copy of DBB zAppBuild

After installing the DBB 2.0 toolkit on USS, we started to update our customized version of zAppBuild.

Upgrading to DBB 2.0 requires updating your existing build framework implementation due to changes to the DBB APIs. With the removal of the DBB WebApp, the build scripts need to use the MetadataStore package (com.ibm.dbb.metadata.*) rather than the RepositoryClient package (com.ibm.dbb.repository.*, which is only available with DBB 1.x). Additionally, if you have not yet switched to using the SearchPathDependencyResolver and SearchPathImpactFinder APIs introduced in DBB 1.1.2 for dependency analysis and impact analysis, respectively, then this migration step is the right opportunity to do so, as DBB 2.0 drops support for their deprecated API counterparts (DependencyResolver and ImpactResolver). The new search path configurations improve the user experience, but may also impact how existing applications have configured the build framework.

Let's focus on the required updates of the build framework implementation. With the release of DBB 2.0, both of DBB's public GitHub repositories released new versions to use the latest DBB 2.0 APIs. The below table provides links to the latest versions.

GitHub Repository	Release
DBB Community Repository	DBB Community v2.0.0
DBB zAppBuild Repository	zAppBuild v3.0.0 onward

It is important to note that only v3.0.0 and onward of zAppBuild is compatible with DBB 2.0. For the public repository of zAppBuild, a new branch with the name zAppBuild_2_x was created to continue serving DBB 1.1.x users.

In our GitLab environment, we manage our team's version of zAppBuild, which is very close to what is available on the public repository, IBM/dbb-zappbuild. For our team's environment, because we intend to move all applications to DBB 2.0 and don't expect any further updates for our pipelines using DBB 1.1.x, we decided to only maintain one branch for zAppBuild: main. If you plan to keep some pipelines working with DBB 1.1.x, you can create a zAppBuild_2_x branch for this.

Our CI/CD pipelines typically clone the main branch of our zAppBuild repository. Therefore, when upgrading from zAppBuild v2.x to v3.0.0, we needed to be careful of how we updated our main branch. We decided to use Git tags to indicate the different versions. Before making any updates to main, we first created a Git tag, 2.5.0, from the main branch in order to remain compatible with DBB 1.1.3. Then, to avoid breaking pipelines working with DBB 1.1.3 while we updated the main branch to be compatible with DBB 2.0, we required existing pipelines to check out the 2.5.0 tag rather than the head of the main branch.

To update our own clone of zAppBuild managed in GitLab, we performed the code updates on a feature branch by following the process described in DBB zAppBuild Introduction and Custom Version Maintenance Strategy.

The updates that were introduced in zAppBuild version 3.0.0 are about providing support for establishing the metadata store connection, which is now directly connecting to the database via JDBC. The updates introduced in zAppBuild version 2.5.0 are for transitioning to the SearchPathDependencyResolver and SearchPathImpactFinder APIs. These changes were delivered to zAppBbuild via the zAppBuild v3.0.0 release and zAppBuild v2.5.0 release. You may want to check out the changelog for all the updates. Although the changes sound significant for DBB 2.0, all we need to do is update the references to the RepositoryClient package to refer to the new MetadataStore package instead. It is important to understand that all the methods against the DBB store are kept the same, which saves a lot of work.

To test the updated build framework implementation, we used the provided MortgageApplication sample application along with the file-level metadata store implementation in DBB 2.0 to avoid any significant infrastructure changes. We reconfigured the GitLab pipeline for the MortgageApplication to use our feature branch containing the planned updates to validate the changes before merging them back to the main branch of zAppBuild.

When updating your copy of zAppBuild, you can go ahead with the most recent release of zAppBuild.

Prepare our GitLab pipelines for DBB 2.0

In our GitLab setup, pipelines can be defined with explicit GitLab CI/CD pipeline definitions or templates, but we noted that all our pipelines contain the following common steps:

Running a downstream pipeline to check out our zAppBuild framework,
Invoking DBB and zAppBuild for the Build step,
Packaging using the sample from the pipeline scripts in IBM/dbb, which we have stored in a shared location on USS,
Triggering UCD deployment using the sample provided by IBM/dbb,
Cleanup activities such as dataset cleanup and metadata cleanup (that is, removing unused collections and build groups).

Our goal was to prepare the CI/CD pipelines of the applications for a smooth migration without any required downtime or maintenance window.

The next sections discuss the approach that we took to meet our goal, which can be broken down into these steps:

Enable GitLab pipelines to specify the DBB version-specific environment variables

Our pipelines contain multiple steps that use groovyz to execute Groovy scripts.

In the past setup of our GitLab environment, the DBB version was resolved through the GitLab Runner's environment variables, typically making use of a DBB_HOME variable. The DBB utilities were directly cloned to a specific location on USS, containing a copy of the pipeline utilities for the packaging, deployment and cleanup steps. The path to these DBB Utilities was referenced in pipeline definitions by a local variable called DBB_EXTENSIONS.

To enable a pipeline to use DBB 2.0, we need to set a different value for the DBB_HOME variable. Additionally, some DBB utility scripts need to run with Java 11 and must be updated to their latest versions. This is because DBB 2.0 no longer ships the Apache HTTP client that was previously used in these DBB utility scripts. Therefore, we decided to store the new version of the DBB utilities in a different location on USS, in order to avoid any conflicts with the DBB utilities running for Java 8 and DBB 1.1.x for our current pipeline configurations.

Rather than modifying multiple references in all our pipeline scripts, we decided to introduce a new setup script, called dbbsetup.sh, to be centrally managed in a configuration directory in USS. This setup script will provide more flexibility for future updates.

The dbbsetup.sh script evaluates a new environment variable named DBB_ENV, which we created for our own use. This environment variable will be set by each GitLab pipeline to specify which version of the setup should be used. Passing a value as an environment variable can also be done with any other pipeline orchestrator, which makes our recipe easily reproduced with other setups.

Based on the value of the DBB_ENV environment variable, the dbbsetup.sh script exports various environment variables to define a consistent execution environment for DBB, Java, and the DBB utilities. As shown in the following script, if the environment variable DBB_ENV is set to DBB20, then the associated environment variables are set to point to DBB toolkit version 2.0, the DBB pipeline utilities of DBB 2.0, and to define the credentials to connect to the Db2z metadata store. (The METADATASTORE_CREDENTIALS variable will be further explained in later sections of this blog post.) If the environment variable DBB_ENV is set to any other value or does not exist, then the associated environment variables will point to the existing configuration for DBB 1.1.x.

#! /bin/sh
#
###############################################################################################
##  dbbsetup.sh: Setup shell script to set DBB Environment Variables
##
###############################################################################################

if [ -z $DBB_ENV ] ; then
  echo "!* Warning environment variable DBB_ENV not set."
fi

if [ "$DBB_ENV" = "DBB20" ] ; then
  echo "-- Setting environment variables for DBB 2.0 --"
  export DBB_HOME=/usr/lpp/dbb/v2r0/
  export DBB_CONF=/usr/lpp/dbb/v2r0/conf
  export DBB_JAVA_HOME=/usr/lpp/java/J11.0_64/
  export PATH=$DBB_JAVA_HOME/bin:$PATH
  export DBB_EXTENSIONS=/var/dbb/extensions/dbb20
  export METADATASTORE_CREDENTIALS="--id GITLAB --pwFile /var/dbb/config/db2-pwd-file.xml"
else
  echo "-- Setting environment variables for DBB 1.0 --"
  export DBB_HOME=/usr/lpp/dbb/v1r0/
  export DBB_CONF=/usr/lpp/dbb/v1r0/conf
  export DBB_JAVA_HOME=/usr/lpp/java/J8.0_64/
  export PATH=$DBB_JAVA_HOME/bin:$PATH
  export DBB_EXTENSIONS=/var/dbb/extensions/dbb
  unset  METADATASTORE_CREDENTIALS
fi

Because all the existing pipelines were still leveraging environment variables set by default at the GitLab Runner level, it was not necessary to modify them to introduce the call to the dbbsetup.sh script and the DBB_ENV environment variable just yet.

Set up security for DBB 2.0 for the pipeline user on z/OS

The required changes from the RepositoryClient to the MetadatastoreFactory API also modifies the way the build framework connects to the DBB metadata repository, which contains the dependency information and the build result history, as well as the evaluation of the user's DBB role.

Previously with DBB 1.1.x and zAppBuild 2.x, the supplied credentials were used to connect to the DBB WebApp and to map the user to a DBB role using either LDAP or the basic user registry. In contrast, DBB 2.0 now validates the DBB role of the user executing the build framework when establishing the JDBC connection for the Db2 metadatastore against RACF. Therefore, the pipeline user, which is running the scripts that store dependency information and build results, is required to be at least a member of the DBBUSERS group.

Our GitLab Runner executes the pipeline tasks under the GITLAB RACF user. To assign a DBB role to our user GITLAB, we added it to the DBBUSERS RACF group. A sample job for the required RACF definitions is planned to be provided in an upcoming DBB release. You can follow the idea DBB-I-34 on the IBM Ideas portal to get updates on this sample job. It is important that the groups specify an OMVS segment and a GID. Please reach out to support for any required assistance.

The credentials used by the MetadatastoreFactory to create the JDBC database connection are supplied to zAppBuild via one of the following CLI arguments when invoking build.groovy:

--id GITLAB --pwFile /var/dbb/config/db2-pwd-file.xml
--id GITLAB --pw ****

While previously, the credentials could be stored in a properties file within zAppBuild, they can now only be passed via the CLI. It is recommended to store the credentials in the secret store of the pipeline orchestrator. Please note, that the password for the JDBC connection needs to be encrypted with the DBB Db2 password file utility and the user needs be granted permissions to the DBB database.

In our case, the pipeline user executing the scripts is the same one used to create the JDBC connection to the DBB database.

Update existing application pipelines

As mentioned earlier, all the existing pipelines are required to update their pipeline definitions as part of their current release cycle to include the call to our dbbsetup.sh setup script to prepare for switching to DBB 2.0. In our case, there were some prerequisite steps, such as updating the main branch of our zAppBuild repository with the updates for DBB 2.0.

Check out the specific zAppBuild Git tag

An important, required change was needed for all existing pipelines that continue to run with DBB 1.1.3. As zAppBuild is updated to its latest version (3.0.0 and above) which is not compatible with DBB 1.1.3, all existing pipelines had to clone the 2.5.0 tag of our custom zAppBuild implementation.

In our GitLab setup, zAppBuild is retrieved for each build through a downstream pipeline. The trigger section of the pipeline definition had to be changed accordingly, to use the Git tag to identify which version of zAppBuild must be retrieved to USS (example shown in the following snippet). Depending on where this step is defined, either in the pipeline or in a template, the syntax may vary.

In our situation, all application pipelines were required to complete this prerequisite before we could proceed with merging any DBB 2.0-related updates into the main branch of our team's zAppBuild repository.

zAppBuild-Update:
    stage: Import
    needs: ["Preparation"]
    inherit:
        variables: false
    variables:
        CI_BUILDS_DIR: "/u/gitlab/gitlab-runner/zos/builds"
        GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_PROJECT_NAME/$CI_COMMIT_REF_NAME/dbb-zappbuild
    except:
        variables:
            - $CI_COMMIT_TAG != null
    trigger:
        project: DAT/dbb-zappbuild
        branch: 2.5.0
        strategy: depend

Include the setup script

We updated our pipelines' definitions and existing templates to invoke our new dbbsetup.sh script. This additional script is required for all jobs that invoke a script through groovyz - in our case, those were our build, code review, packaging, deployment and housekeeping stages. In the following snippet for our Build and Test stage, you can see that the first script we call is now dbbsetup.sh:

   ...
Build and Test:
    stage: Build and Test
    tags: [z/OS]
    needs: ["Preparation", "zAppBuild-Update"]
    variables:
        GIT_STRATEGY: none
        GIT_CHECKOUT: "false"
    coverage: '/Global Code Coverage Percentage: \d+/'
    artifacts:
        name: "report-${CI_PIPELINE_ID}"
        when: always
        paths:
            - "$CI_PROJECT_DIR/BUILD-OUTPUT/*.log"
            - "$CI_PROJECT_DIR/BUILD-OUTPUT/*.html"
            - "$CI_PROJECT_DIR/BUILD-OUTPUT/*.json"
            - "$CI_PROJECT_DIR/BUILD-OUTPUT/*.xml"
            - "$CI_PROJECT_DIR/BUILD-OUTPUT/*.pdf"
            - "$CI_PROJECT_DIR/BUILD-OUTPUT/*.txt"
            - "$CI_PROJECT_DIR/BUILD-OUTPUT/*SONARQUBE/*.*"
        reports:
            junit: "$CI_PROJECT_DIR/BUILD-OUTPUT/*.xml"
            cobertura: "$CI_PROJECT_DIR/BUILD-OUTPUT/*CCCOBERTURA/*.xml"
    except:
        variables:
            - $CI_COMMIT_REF_NAME =~ /^rel*/ && $CI_COMMIT_TAG != null
    script:
        - source /var/dbb/config/dbbsetup.sh
        - export LLQ=`echo ${CI_COMMIT_REF_NAME} | sed 's/[^a-zA-Z0-9]//g' | sed 's/^[0-9]*//g' | cut -c 1-8 | tr '[:lower:]' '[:upper:]'`
        - export shortRefName=`echo $CI_COMMIT_REF_NAME | cut -c 1-3`
        - $DBB_HOME/bin/groovyz $ZAPPBUILD/build.groovy $METADATASTORE_CREDENTIALS --workspace $CI_PROJECT_DIR --hlq ${HLQ}.${LLQ} --workDir $CI_PROJECT_DIR/BUILD-OUTPUT --application $applicationName --logEncoding UTF-8 --impactBuild -cc --verbose
        - cd $CI_PROJECT_DIR/BUILD-OUTPUT
        - ...

Introduce the `DBB_ENV` environment variable

With the introduction of the dbbsetup.sh script, a new environment variable needed to be added to the pipeline definition to define which DBB setup should be used. Remember, that the script defaults to the environment variables for DBB version 1.x. For clarity in the following snippet, we set DBB_ENV to "DBB10" (indicating DBB 1.x), as we have not yet prepared our application to use DBB 2.0. (That will be covered in Part 3!)

variables:
    DBB_ENV: "DBB10"          ## the new environment variable
    HLQ: "GITLAB.RETCALC"
    APPLICATION_NAME: "retirementCalculator"
    UCD_APPLICATION: "retirementCalculator-GitLab"    
    UCD_COMPONENT: "retirementCalculator-GitLab"

Prepare to pass the new credentials for the Db2 metadatastore

Starting with zAppBuild 3.0 and DBB 2.0, the credentials of the Db2 metadatastore are expected to be passed via CLI rather than a definition in a properties file. To prepare for DBB 2.0, we are leveraging a new, ad-hoc METADATASTORE_CREDENTIALS environment variable during the invocation of zAppBuild framework as shown following snippet.

For an application pipeline using DBB 1.1.3, this environment variable is empty, as unset by the dbbsetup.sh setup script. For the pipelines that are migrated to DBB 2.0, the dbbsetup.sh script will set the appropriate value for it.

...
$DBB_HOME/bin/groovyz \
 $ZAPPBUILD/build.groovy \
 $METADATASTORE_CREDENTIALS \
 --workspace $CI_PROJECT_DIR \
 --hlq ${HLQ}.${LLQ} \
 --workDir $CI_PROJECT_DIR/BUILD-OUTPUT \
 --application $applicationName \
 --logEncoding UTF-8 \
 --impactBuild
...

Conclusion

In this blog entry, we have detailed the preparation steps that we took to facilitate a smooth migration. This included the updates to our team's custom version of zAppBuild, the introduction of the setup script, and the changes required for existing pipelines. Yet, no application has migrated to DBB 2.0, as this will be performed in the next blog post, Part 3 - Migrate z/OS applications to adopt DBB 2.0!

Authors of this post

Lauren Li, DevOps Acceleration Team

Mathieu Dalbin, DevOps Acceleration Team

Dennis Behm, DevOps Acceleration Team

Development and Pipeline - Group home