Why leveraging WCA for unit test? or test generation?
Verification and validation of a software product are crucial for its success. A robust set of tests and verification metrics builds confidence among developers, testers, reviewers, and other stakeholders, ultimately reducing potential issues in production. There are many testing frameworks available for different programming languages, and now there are code assistants augmenting developers' and testers' ability to write more efficient and effective tests. This article focuses on how leveraging IBM® watsonx™ Code Assistant (WCA) can help write effective tests more quickly and improve code and test coverage.
What’s WCA?
WCA is a purpose-built, generative AI-assisted product designed to accelerate both code and content generation. It offers a wide range of capabilities, including code generation, code explanations, unit test creation, AI-derived code documentation, and many more. WCA can be seamlessly integrated into popular IDEs like Visual Studio Code and Eclipse, making it accessible through conventional chat interactions. It supports a variety of programming languages, such as Go, C, C++, Java, JavaScript, Python, TypeScript, and more, acting as a reliable coding companion. In this article, we'll explore how leveraging these features can significantly boost code and test coverage.
Prompt engineering with WCA
Prompt engineering is a powerful technique for generating new content, and WCA leverages this to perform variety of tasks. In addition to regular prompt support, WCA has optimized prompts for specific tasks. This article focuses on how utilizing both can enhance coverage.
The WCA chat interface, when integrated with VSCode editor.
Fig. WCA chat interface
WCA for code coverage
Code coverage ensures each line of code is tested at least once and can be measured in terms of percentage. Developers write unit tests against source code to achieve this. They can write unit test which gives coverage of 50 to 60% easily but taking coverage beyond is challenging and time consuming. Depending on criticality of product this percentage can be targeted.
Let’s explore how WCA can help in code coverage -
a) Using chat interface (prompt engineering):
In WCA chat interface we can pass query prompt and context information for unit test generation. For example –
List possible unit tests using python unittest module in-order to have 100% code coverage for below piece of code
<Your source code> <ENTER>
Result: <List of unit test methods>
Can ask unit tests using another framework available
I need implementation of these tests using python pytest module
Can translate scenario into methods
I need implementation of these test scenarios using python unitttest module
Can ask additional tests
Can you add some additional edge cases?
And so on…
b) Using optimized prompts:
WCA provides optimized prompts for unit test generation. For exact usage, refer to this page. These optimized prompts reduce the effort of prompt engineering and gives flexibility to users to write effective unit tests using various testing frameworks available in different programming languages.
Syntax:
/unit-test [using <test_framework>] <code reference> [similar to (<unit test reference>)] [additional instructions]
Let’s explore this syntax with python built-in unittest testing framework.
To generate unit test for a given method or class
/unit-test using unittest @collect_pod_logs
Generate unit test similar to another test
/unit-test using unittest @collect_pod_logs similar to @test_delete_job_not_found
Provide additional context to prompt
/unit-test using unittest @collect_pod_logs similar to @test_delete_job_not_found cover missing edge cases
etc.
c) Using Editor CodeLens feature
This feature helps to generate unit test for a class/method with a mouse click
d) Copy block of code in WCA chat window
With help of this feature we can move selected code block to chat window for unit test generation
By adopting different techniques outlined above, we can enhance code coverage to an optimal level. For instance, from 75 to 95%.
Code Coverage report
Before -
| Name | Stmts | Miss | Cover |
|---------- | -------: | -------: | ------: |
| job.py | 191 | 47 | 75% |
After -
| Name | Stmts | Miss | Cover |
|---------- | -------: | -------: | ------: |
| job.py | 191 | 10 | 95% |
Though WCA does not provide the coverage report directly, it helps by providing installation steps for the necessary tools and increasing the unit test count, thereby boosting the coverage.
The above coverage report was generated using the Python coverage tool, showing statement coverage. More complex coverage metrics, such as branch and path coverage, can be targeted to achieve more comprehensive verification objectives. And for such goals AI assisted tools like WCA are very helpful.
WCA for test coverage
Test coverage indicates the percentage of requirements covered by at least one test case.
Let’s make use of WCA chat functionality (prompt engineering) to generate test scenario/test cases from a product requirement document. For example –
Can you give me test cases for below feature requirements?
<Your PRD text> <ENTER>
Result: Depending upon context it can list methods or test scenarios
Next prompt:
List possible test scenarios in-order to increase test coverage
Result: listed 10 test scenarios
List possible test scenarios in-order to have 100% test coverage for below feature requirement
<Your PRD text> <ENTER>
Result: listed 20 test scenarios
List missing edge cases
Result: listed edge scenarios
And so on…
0: None, 1: Indirect, 2: Direct
Again, WCA is helping in adding test cases into requirement traceability matrix, thereby increasing the test coverage.
Interesting coverage aspects to look at from traceability matrix are highlighted in red:
- Vertical red column has values for depth of coverage for a particular requirement.
- Horizontal red row depicts the breadth of the coverage for a test case, or in other words shows its importance.
- And, the no test case row causing overall test coverage percentage to 88.89.
Summary
In this article, we explored several ways to generate unit tests and derive test cases/scenarios from the product requirement document, demonstrating the usefulness of IBM watsonx Code Assistant (WCA). We also examined the coverage metrics for each of these using sample examples, and noting the improvements made. Making use of prompt engineering with WCA not only enhances the coverage metrics but also increases the efficiency of developers and testers, giving them more time to focus on other quality and security aspects of the product.
If you're interested in exploring WCA, please refer to the provided links.
References:
