We can see that we have a new issue that was opened by one of our testers.   In GitLab you can see an issue list and also leverage Issue Boards.   Here is the list view of issues:

You can also easily create your own Kanban boards and customize them like the “Project Sprint Planning” board I created for our team as shown below:

Creating the Merge Request

As we inspect the details of the new issue, we see that the “liveness” API is returning a DOWN status and this impacting the overall front-end system. 

Notice in the right-hand pane that an issue can be associated to an epic, milestone, and you can provide estimates for time tracking, specify weights, as well as assign labels like “severity::2”.

After reviewing this defect, we decide that we will begin to address the issue by creating a Merge Request.  To do this you, will click the “Create merge request” button as shown above.   Another dialog will appear that allows you to confirm the owner, assign approvers, specify merge request dependencies and other options as shown below:

One of the nice features in GitLab for IBM that you can leverage is a “Vulnerability-Check Rule”.   Using this rule, GitLab can automatically assign a reviewer should vulnerabilities trigger the rule condition.   Security is an important feature to keep in mind when developing cloud-native applications.

By clicking the "Create merge request" button from this confirmation page, a new “feature branch” will be created in git which allows the developer to work in isolation to correct the bug in the program.

Using the Liberty Dev Dashboard in our IDE of Choice

As a Liberty developer, a valuable capability that you can use for your development and testing of your code is the Liberty Dev Dashboard.  For this blog, I am using the VS Studio Code IDE and have installed the “Open Liberty Tools” extension.  This capability is often referred to as “dev mode” and it helps to boost developer productivity with features highlighted below:

As I begin to work on my feature branch in my IDE, the "Liberty Dev Dashboard" has detected my Liberty-based application and gives me options to start my application in a container on my developer workstation as shown below:

This process will build my code, create a container, and then start the container in “dev mode”.   Using the “dev mode” features, I can run unit tests (by selecting the "Run tests" pop-up menu item as shown above) and view the results of those tests.   Here are the results I observed:

Sure enough, my unit test to check the response from the liveness endpoint is failing just as the issue had described.    I can even open a local browser and connect to the web application running in the container and see the same failure as well.

To correct the issue, I will edit the SystemLivenessCheck.java and fix the bug found on line 39 as shown below:

After fixing this code, I also make a change to the web console background color and font size as well as update some text in the index.html page to highlight our code changes.    Since I am a WebSphere fan, using a purple color makes a lot of sense for this demo.

After saving these changes, the Liberty Dev Dashboard will automatically update our running container for us.   There is no need to stop, re-compile and start the container.   We can simply run our unit tests once more to confirm they are working as shown below:

We can then commit our code changes to our feature branch.

The GitLab CI/CD Pipeline and Security Scanning

At the moment you commit your code changes to the feature branch, a GitLab CI/CD pipeline will begin to run.   This pipeline will include the proper phases and steps within each phase to ensure DevSecOps best practices are leveraged early and often. 

Return to your merge request and notice that a pipeline is in progress as shown below:

As we examine the running pipeline, we see that the application is being built using a GitLab Runner that comes pre-packaged with cloud-native build capabilities to support maven and gradle which are popular for Liberty-based applications.   GitLab can detect the runner needed for your project automatically using the “Auto DevOps” feature.

When the build completes, the pipeline will create a docker container as shown above in the Build phase and then store the container in the GitLab Container Registry.   You can see what is stored in the container registry by clicking on "Packages & Registries" in the left-hand pane and then selecting "Container Registry":

You can leverage other container registries as well if you prefer.   Our pipeline will then run a container scan as shown below:  

A container scan is a very important step for new cloud native applications.   Your application’s Docker image may itself be based on Docker images that contain known vulnerabilities.   The scanning feature can help you ensure your containers are secure and alert you of potential vulnerabilities right away. 

You can also see at the same time the application is being compiled, our pipeline has run some built-in scans that come with GitLab for IBM as highlighted below:

• Dependency Scans – this will look for vulnerabilities in libraries used in your project (they can be open source libraries or perhaps shared libraries you are using)
• License Scans – if you are using open source projects, the license scans can be setup to make sure you do not have any license violations
• Static Application Security Tests (SAST) Scans – this is white-box testing that can look for items like a “sql scripting injection” in your code as an example
• Secret Detection – checks to see if you are exposing secrets
• Coverage-guided Fuzz Testing – using this technique, a fuzz test sends random inputs to an instrumented version of your application in an effort to cause unexpected behavior.
• API Fuzz Testing – using this technique, a fuzz test will set operation parameters to unexpected values in an effort to cause unexpected behavior and errors in the API backend. This helps you discover bugs and potential security issues that other QA processes may miss.

As the pipeline continues to run, you can incorporate performance or load tests as shown below using some built-in GitLab capabilities:

Gitlab uses k6, a free and open source tool, for measuring the system performance of applications under load.   IBM offers additional performance testing capabilities with our IBM DevOps solutions that you can also integrate into your CI/CD pipeline.   This includes Rational Performance Tester, Rational Test Virtualization for service virtualization, and Rational Test Automation Server (RTAS) as examples.   Testing can be the number one bottleneck in the development process (functional, performance, security as examples).   With GitLab for IBM, we can help address this to help you accelerate the delivery of your applications with better quality.

Finally, we will execute Dynamic Application Security Testing (DAST) against the running application in the REVIEW environment as shown below:

The DAST tests are a form of black box testing that can scan the running application to look for issues.   Using GitLab for IBM, DAST can analyze the application in two ways:

• Passive Scan Only (DAST default). DAST executs OWASP Zed Attack Proxy (ZAP) Baseline Scan and does not actively attack your application.
• Passive and Active Scan. DAST can be configured to perform an active scan to attack your application and produce a more extensive security report.

Our pipeline ends at a manual step to deploy the application into a Quality Acceptance environment which can be leveraged to validate the environment alongside other applications.   Before we perform that deployment, we will return to the merge request to view the results of what has been executed.

The Merge Request and Collaboration

After the pipeline completes against our feature branch changes, we can now return to the Merge Request and view a lot of information to help us make an informed decision as to whether we should proceed with a merge of our changes back into the main line.

Using the “View app” button as shown above, you can interact with your REVIEW application to see a “live preview” of the changes you have made.   This allows designers and product managers to easily see your changes and collaborate with you.   A new tab will be opened for you so you can interact with the web application:

We can also validate that our REST API is working as expected in this same REVIEW app as shown below:This looks great so far.  Let’s continue to evaluate the findings from the DevSecOps tests that were performed as part of our pipeline.

The GitLab Merge Request is where the development team can also collaborate with operations, the security team, and other product stakeholders.   Each section in the merge request has a specific purpose as highlighted below:

In the past, you may have had to go to separate point solutions to view results of code quality scans, license scans, and security scans.   In GitLab, you can view all the information in one location without having to perform context switches.   There is real value in leveraging GitLab for IBM by having a single interface, single data store and single conversation from a DevOps perspective.

Let’s drill into some of the security scan results by clicking the “View full report” button as shown above:

For each potential vulnerability identified, you can use the buttons to the right to view more information, create a new issue which is populated with the details about the vulnerability, or dismiss the issue should the team feel it is not a problem.

With GitLab for IBM, the development team and security team collaborate to “shift-left” their testing efforts and work together to address high or critical issues the moment they are found with each code commit.   Many organizations have a single CSO team that generally catches security issues right before the PRODUCTION go-live.   This is not a very efficient approach and many times it can be difficult to understand the context around changes and who should address the issues.   Using GitLab for IBM, your development resources can become extended members of the security team and help you secure your environment.   You can develop “security champions” in the organization that assist the application teams and work alongside the main security team.

NOTE: GitLab for IBM allows our clients to run unlimited scans where other solutions will typically charge for each scan.   This can be very costly which is why some teams wait until they get to PRODUCTION to run their security scans.

If you are on a team that is just starting the journey of incorporating security tests for a project, the first time you leverage this pipeline you may find that the GitLab security testing results will tend to be “overprotective”.    GitLab for IBM provides the tools to make it much easier for your   Security CSO team and the development team to collaborate and work through any technical debt in this area.    We find that being “overprotective” is better for establishing a good working baseline, especially when coming from a place of no security scans.

Using the “Security & Compliance” menu found in left panel, you can review the “License compliance”, “Dependency list” and the “Vulnerability report” details as shown below:

Having these security capabilities integrated into your development process early and often is so very valuable.  Using GitLab for IBM, you can easily identify, collaborate, and resolve vulnerabilities quickly in an agile way. 

Merging Changes Back into the Main Branch

Once you are satisfied with your test results and your updates, you can mark the merge request as being ready.   As the developer, I will click the “Mark as ready” button as shown below:

You may have a software leader review the changes that has the privilege to merge the changes.   When ready, the software leader clicks the “Merge” button as shown below to initiate the merge of your feature branch changes into the main branch:

This action will initiate the execution of another pipeline that will run against the main branch.   If we return to the Project view, we can see that it is running as shown below:

This pipeline can be configured to have some of the same steps to build the application, create the container, and run security tests. 

Deployment into a Kubernetes Cluster in OpenShift

When the GitLab CI/CD pipeline runs against our main branch, the “deployment” step of the process will involve a deployment of our container into an OpenShift (or Kubernetes) cluster.  

A best practice for Liberty-based container deployments is to leverage the Open Liberty Operator.   You can install the Open Liberty Operator from OperatorHub for use on Kubernetes or OpenShift.   The operator is also available as a Red Hat-certified operator from OpenShift Container Platform (OCP). 

We have installed the “Open Liberty Operator” which simplifies deployment of our containers into an OpenShift cluster.   You can see the Operator installed in our OpenShift environment as shown below:

This operator will watch Open Liberty resources and compare the current state of resources to the state you desire.   When a discrepancy exists, the operator creates, updates, or deletes Kubernetes resources to return the application back to the proper state.

To initiate the deployment leveraging the operator, our GitLab pipeline calls IBM UrbanCode Deploy to trigger the deployment to the “INTEGRATION-OpenShift” environment as shown below:

The IBM UrbanCode Deploy process will deploy the new container into a Kubernetes cluster:

NOTE: in a future blog post, we will explore the built-in GitLab capabilities for cloud-native deployments in a Kubernetes cluster.   IBM UrbanCode Deploy is a very popular solution for complex, on-premise deployments alongside newer cloud-native deployments.

In the OpenShift console, we can now see that the application is deployed in the Workloads -> Deployments view as shown below:

You can view more details about the deployment by clicking on the name column:

You can also see that the operator helped us create a network route to the application in the OpenShift cluster as shown below:

By clicking the location link for the network route, we can observe the running application inside OpenShift:

Wonderful!   We were able to successfully resolve the issue and do it quickly with the security testing safeguards in place.

Summary

As you have seen in this blog, DevSecOps is essential for every development project.   You have seen in this blog post that GitLab for IBM can provide a complete DevOps platform which is unified and backed by a single data store. 

Success can sometimes be defined as deploying ahead of schedule.   That is not entirely true as most advanced software applications will fail without proper security.   Moving fast without taking security into account can lead to costly and time-consuming efforts which inevitably can push back the software release date.

Implementing security throughout the entire development (and delivery) process allows developers to resolve smaller issues before they become large and more cumbersome to resolve.   Performing security scans at the end of your process can make it more complicated and inefficient to correct serious issues.  

You have observed in this article how GitLab for IBM can incorporate DevSecOps best practices early and often in your Liberty development process at time of code commit.   If your team is not implementing security from the start of the project, it is time to get on board with DevSecOps.   Implementing Liberty-based cloud native applications using the combination of GitLab for IBM and some excellent IBM DevOps Tooling can help you achieve your goals of delivering quality software that is secure and at a much quicker pace than you have in the past.