Code coverage is a valuable quality control tool used to understand how well your z/TPF applications are being tested. Code coverage reports provide information about what lines and percentages of lines of your z/TPF applications are being executed.
PJ46334 enhances the code coverage REST services to provide a script-friendly workflow for code coverage collection and reporting for the z/TPF platform. You can call services from your test framework to generate reports for your z/TPF applications. The code coverage reports allow developers and managers to better understand how completely their z/TPF applications are being executed by the test cases and provide insight into test case quality.
To better illustrate this, let's consider the role of the application developer—we'll call him Zach. Zach wants to see how much of his new code is being covered by his test cases. First, Zach issues a start request to register his program for code coverage collection. Zach runs his test cases. After his last test case is executed, Zach issues a stop request with the native formatting parameter set to true and the format parameter set to his desired report format. The stop service will stop the code coverage collection and produce a report on the
mounted z/TPF file system in the format specified in the request. By issuing status requests, Zach identifies when the code coverage report has been produced and then retrieves his code coverage report from the z/TPF file system by issuing a GET request. Once the report is downloaded, Zach runs the delete service on the code coverage report to free space on the z/TPF file system for other users.
See the
z/TPF scriptable code coverage support documentation in the z/TPF IBM Knowledge Center for more information, including examples that demonstrate how to call the code coverage REST services on z/TPF.
PJ46334 also adds the Cobertura format to the list of code coverage results formats that you can choose from. Cobertura format results can be viewed in any code coverage viewer that supports the Cobertura format such as a SonarQube server. The Cobertura format was chosen so that it can be imported into the free version of SonarQube Community Edition with "SonarQube C++ plugin (Community)".
Code Coverage for new and changed code with SonarQube
With the Clean as You Code approach to code quality, your focus is always on new code, which is the code that was added or changed from your previous version. As a developer, your priority is making sure the code you write today is clean and safe.
To view code coverage results for the new and changed code for your z/TPF project, you must first run a baseline scan and create the project in SonarQube. SonarQube identifies new and changed code by comparing the baseline scan and any scans that follow.
We have provided some sample Python scripts to help a developer who is working on a z/TPF project to view code coverage results for new and changed code in SonarQube. Download the sample scripts from
TPF Family Products: Drivers website and follow the instructions in the ReadMe to install and configure them.
The following example, following the role of a systems developer (we'll call her Sophie), gives you an idea on how to use the sample scripts for creating the baseline for your z/TPF project, running code coverage, and then running a scan to import the Cobertura formatted code coverage results into SonarQube. SonarQube Community Edition version 8.4.2 with SonarQube C++ plugin (Community) was used for this example.
- Sophie creates a z/TPF project named PJ46334 in /home/sophie on the Linux on Z system, with the following directory structure and source in it:
PJ46334
|--build
|--appl
|--tpf
|--base/rt
|-- cd00.cpp
|-- cd01.cpp
|-- cd02.cpp
|-- cd03.cpp
|-- cd04.cpp
|-- cd07.cpp
|-- cd09.cpp
`-- cd13.cpp
- Sophie then copies the tpf_run_native_ccv.properties file into the project directory from the scripts install location /ztpftools/tpfNativeCodeCoverage.
cd /home/sophie/PJ46334/
cp /ztpftools/tpfNativeCodeCoverage/tpf_run_native_ccv.properties ./
- Sophie updates the tpf_run_native_ccv.properties file with the values for the z/TPF test system and TPF project specific variables below.
tpfhost = 9.57.13.214
tpfport = 81
programs = CD00;CD01;CD02;CD03;CD04;CD13;CD14
subsystem = BSS
testcaseName = CCV_test
outputDir = sophie
- Sophie performs a baseline Sonar scan on her project
cd /home/sophie/PJ46334/
/ztpftools/tpfNativeCodeCoverage/tpf_scan_project.py
- She sees the following output:
>>> Initializing the git repository for the project /home/sophie/PJ46334
>>> Running sonar scan...
11:44:45.891 INFO: ANALYSIS SUCCESSFUL, you can browse http://tpftk1.rtp.raleigh.ibm.com:9000/dashboard?id=PJ46334
All done
- Sophie navigates to the SonarQube dashboard by following the link provided in the output from the baseline scan. At this point the SonarQube dashboard shows that project PJ46334 exists, but does not have any code coverage data, because Sophie has made no changes to the project yet.
- On Linux on Z, Sophie updates the source code and adds a new source file, cd14.cpp . She builds the project with debug information, and loads to the z/TPF system using MakeTPF tools
- On Linux on Z, Sophie starts code coverage by issuing the following commands:
cd /home/sophie/PJ46334/
/ztpftools/tpfNativeCodeCoverage/tpf_run_native_ccv.py start
- She sees the following output:
>>> Starting Code coverage session on the TPF test system...
All done
- Sophie invokes the code that was just loaded to z/TPF by running tests
- On Linux on Z, Sophie stops code coverage by issuing the following commands:
cd /home/sophie/PJ46334//ztpftools/tpfNativeCodeCoverage/tpf
_run_native_ccv.py stop
- She sees the following output:
>>> Stopping code coverage on the TPF test system...
Code coverage session has been stopped and the results file /TPFCodCov/formatted/sophie/CCV_test_20210208131015.xml has been created.
All done
- On Linux on Z, Sophie downloads the Cobertura formatted code coverage result file from the z/TPF system.
/ztpftools/tpfNativeCodeCoverage/tpf_run_native_ccv.py get
- She sees the following output:
>>> Sleeping for 20 seconds to allow for the stop code coverage post processing to complete
>>> Downloading the code coverage results file from TPF...
Code coverage results file /home/sophie/PJ46334/coverage.xml has been downloaded.
All done
- On the Linux on Z system, Sophie performs a Sonar scan on her project
/ztpftools/tpfNativeCodeCoverage/tpf_scan_project.py
- She sees the following output:
>>> Running sonar scan...
14:47:30.034 INFO: ANALYSIS SUCCESSFUL, you can browse http://tpftk1.rtp.raleigh.ibm.com:9000/dashboard?id=PJ46334
All done
- Sophie checks the SonarQube dashboard for her project PJ46334.
- She notices that the quality gate status for the new code shows as "Failed" since the tests she ran covered only 38.86% of the new code and is less than the required 80%.
Note: The default built-in Sonar way quality gate was used for this project. A quality gate is a set of measure-based, Boolean conditions. It helps you know immediately whether your projects are production-ready. Ideally, all projects will use the same quality gate. Each project's quality gate status is displayed prominently on its homepage. You can create your own quality gate in SonarQube and adjust it to meet your needs.
Coverage percentage for Sophie's overall code shows as 19.6%.
Coverage information for each source file in Sophie's project.
Coverage information for the source cd04.cpp.
- Sophie adds more test cases to her test plan and repeats the code coverage and scan until the project has passed the quality gate conditions.
It is important to ensure quality code is being delivered with every release. Code coverage is a valuable quality control tool used to understand how well your z/TPF applications are being tested. You should make code coverage part of your continuous integration (CI) workflow.
Your main focus for code coverage should be on new and changed code instead of the overall code that already exists. One of the ways to ensure your new and changed code is being fully covered by tests is to use our code coverage reports with SonarQube as outlined in this blog. Tools such as SonarQube provide visual insights into what part of your new and changed code is covered by tests and help you identify critical parts that still need to be tested, thus helping you fill the gaps in your test suite.
Check out the APAR
PJ46334 download page for additional details. Also, check out the
TPF Family Products: Drivers website for additional details about the sample Python scripts.