The Practical Test Pyramid is a common methodology for implementing a testing strategy. In this article we will consider how this applies to an IBM API Connect solution.
We will explore the following topics:
- Practical Test Pyramid
- Requirements
- Test types
- Unit Test
- Integration Tests
- Consumer Tests
- Automated Testing
1. Practical Test Pyramid
The practical test pyramid developed by Mike Cohn describes initiating lots of quick, low-level, ‘cheap’ tests on the smallest possible ‘unit’ to give greater confidence that end to end flows will work. Whereas, more expensive, time consuming and disruptive test cases should be fewer in number and will demonstrate connectivity with other systems or all unit tests working together.
Figure 1 - The Practical Test Pyramid (Mike Cohn)
In Marin Fowlers publication, The Practical Test Pyramid, he talks about how the categorisation of the tests and the words used to describe them can become conflated and detract from the general aim of creating systems that work. Below is the usage of the pyramid we believe best applies to an API Test Pyramid.
- Unit Test - A unit test denotes the smallest unit of API testing in a Request being sent and a Response received. Both must match service requirements.
- Integration Test - Often known as a ‘Component Test’, the use of Integration as the test ‘keyword’ shows that these tests should focus on APIs calling independent resources such as a database.
- Consumer Test - APIs are created by providers for the use of consumers. The consumer has an interaction with the whole system end-to-end including any user interface, thus consumer testing applies to full end to end tests.
Figure 2 - The test pyramid in the context of IBM API Connect
2. Requirements
It should not be left to the implementer of the API to dream up tests from thin air. They must be based on business requirements. Without this, the whole basis of testing is undermined.
Before any API project begins there should be a clear description of the expected outcomes. What is the project trying to achieve and what is the minimum request and response that must be delivered in order for the API calls to be successful? These expected outcomes are the requirements of the project, without which no tests can be written and thus no APIs can be written.
Requirements state explicitly what is and is not permitted which allows both positive and negative testing to be completed and might be based upon industry and enterprise standards and specifications.
Tests that are written without requirements will either use existing code and implementation as a source of truth or will rely on the developers view of what should be occurring. It is also possible for these tests to be disputed since there is no individual source of truth for what the implementation should be doing to meet business needs.
3. Test Types
a. Unit Tests
What is an API Unit Test?
The primary question when discussing Unit Testing is what makes up the smallest, testable, self-contained component, derived from the requirements. A unit test for an API should not call a backend service, it should not rely on making requests from outside the API Connect domain and should be executable in any simple IBM API Connect environment.
The smallest testable ‘unit’ for API Connect APIs consist of the four components that make an API;
An API has a primary data representation called a resource which is the conceptual mapping to a set of entities. The address of the resource is displayed as a Uniform Resource Identifiers (URIs) which is crucial to making an API call.
Addresses will have tests such as;
1. Does an exposed address return a correct result?
2. Are unspecified addresses correctly handled?
3. Are misspelt addressed correctly handled?
An operation is the method of a REST API that determines what action to perform on the resource. The four main HTTP verbs that define the operation are; GET (read), PUT (update), POST (create) and DELETE (delete).
Operation Unit Tests will make requests to valid addresses to ensure the API only exposes the verbs that are outlined in the requirement. This test is simply about operation exposure.
Simple security tests might also be applicable here like the absence of tokens, client ids or certificates for secure endpoints but is unlikely to include more advanced tests such as OAuth which would be more applicable in integration tests as multiple calls are needed.
Request data - normally refers to a list of headers and the body of data that is passed with an operation to an address. Tests at this level focus on whether the correct data is provided, the correct headers are used, and that incorrect headers or malformed data are handled correctly.
Response data - normally refers to how the resource call will respond to the request. The data usually includes a status code, headers and a body. Tests at this level aim to ensure that valid and expected output is received when certain requests are passed into the system. Additional tests will be towards the handling of both success and failure scenarios.
Figure 3 - Unit Test Topology
Unit Testing GatewayScript fragments
From the API Connect perspective this is the lowest logical ‘unit’ since testing deeper has little benefit. Whilst lower level testing within the API of specific fragments of gateway code logic might provide some assurance, having quickly deployable unit tests that implicitly exercise that same gateway code should give the same level of peace of mind. Testing the inner workings of the API is irrelevant if the correct response is achieved.
Moreover, lower level testing will add extra overhead in order to implement a unit testing codebase and runtime environment which might inadvisably lead developers to test outside of a DataPower runtime. It is important to note that gateway programming model is security hardened and enriched with DataPower-specific functions where you access and manipulate the variables in the API context during execution. As such, any attempt to test fragments of GatewayScript in an independent runtime is likely to be invalid.
Unit testing of APIs in API Connect should only be concerned with the ‘unit’ which in IBM API Connect is a processing a request being made and processing the expected response.
Ideally it is better to do unit tests on a smaller runtime environment (such as using Docker image for DataPower) so that they can be isolated to a single change which would help to build consistent pipeline deployments. Using a shared environment will bring operational challenges.
Unit tests should not connect to real backend systems as they should be able to run independently and using real backends increases the runtime dependencies of the unit tests.
b. Integration Tests
Having determined that the API’s work in isolation, integration tests can be performed to determine if connectivity with other services works and that the results are handled and remain valid for the requirements.
The concept of ‘Contract Testing’ is discussed in ‘The Practical Test Pyramid’ and is applicable in API Connect API testing too. Connecting services have similar request/response testing as performed for APIs.
By mapping expected requests to expected responses each integrated service has a ‘contract’ of integration. Should the way the API behaves change, the providing service should notify the consumers as part of the release.
Integration tests will look to make calls to connected systems such as a database, microservice or load balancing service. Each connection should be handled independently without multiple linked tests across several systems and services. This might mean exposing unit test environments to other systems so they can perform isolated integration tests between the IBM API Connect layer and the singular service it is calling.
These tests should try to call services that have no dependencies on other services. Whilst the API Connect layer has no control of the services they call; it is better, if possible, to call a service whose own backend service is mocked for these tests so that the integration is affected only by the integrated services.
Figure 4 - Mocked Tests
Testing of integration in API Connect is the one to one communication of the service immediately before or immediately after the API unit.
c. Consumer Tests
Also known as end-to-end testing these tests will reach each service of the system as a full flow in the same way consumers will. Having completed unit tests - that cover a wider API base - and integration tests that connect to each connected service, these tests should be limited only to what is expected for the API consumer to complete.
At this stage, independent user who do not know how the API works internally and are not directly involved in development or lower level testing of the component parts will look to use the system as if they are an API Consumer to see if there are issues outside of the current test scenarios in a concept known as ‘exploratory testing’.
Consumer tests might also include application creation, API subscription and API test call through user interfaces. This should include testing of how the system is used by API consumers to ensure the themes, buttons, selection boxes all work as intended in a process known as User Interface (UI) testing.
Whilst UI tests cover a lot of the usability tests, thorough end-to-end testing should be conducted which looks at user ‘journeys’ through the system i.e. creating an application and subscribing APIs, logging into the system and performing a series of linked API calls such as an Oauth2 flow.
Figure 5 - Consumer Testing
Testing at the consumer level looks at the full end to end journey experienced by the consumer. Each flow should behave as the consumer expects including API responses, user-interfaces that may be required and redirection to other services.
4. Automated Testing
It is possible to perform automated testing at each layer of the pyramid with initial testing on as updates to the API definition are pushed to a code repository. Build tests might include;
- JavaScript Linting - checks against a custom set of specifications such as line length
- Test application subscriptions
- Complete run of unit tests
- Complete run of integration tests
- Automatable consumer tests
Following successful completion of the test suite in one environment e.g. development, the artefact can be made ready for combined testing in integration tests where potential changes to multiple APIs can be tested at the integration and consumer level by running the automated tests against the System Integration Environments.
All APIs non-breaking will be pushed (in an automated way) to non-functional tests where they once again follow the automated test procedure before failing performance APIs are removed from the release.
The remaining changes are pushed through into User Acceptance Testing (UAT) environment which also complete the automated tests but with additional manual exploratory test scenarios in the most production like environment before the same process proceeds into Production.
Figure 6 - Automated Testing
5. Conclusion
In this article we have explored how The Practical Test Pyramid can be utilised in the context of IBM API Connect. An overview has been given on the three types of testing; unit testing, integration testing and consumer testing.
It is important to consider what makes up the independent ‘Unit’ of an API which we have described as a Request and Response using the IBM API Connect and IBM DataPower Gateway in isolation.
Additional stages such as the integration testing and consumer testing are simply names that best describe the actions that apply in this context with descriptions following the general rules defined in the Practical Test Pyramid.