Target audience: Architect, Performance engineer, developers
PDF version available at :
Simplified Performance Methodology
Performance testing is an essential aspect of software development and deployment.
It is the process of evaluating the performance of a system or application under various conditions in order to ensure it meets the desired requirements and can handle real-world usage.
The goal of performance testing is to identify bottlenecks and optimize the performance of the system before it is released to customers.
This text provides a broad overview of the field of performance testing, including key concepts, types of tests, and factors to consider when conducting a performance test.
Types of performance tests
There are several types of performance tests, including stress tests, long-run performance tests, and scalability tests.
Stress tests are used to evaluate the behavior of an application under high levels of stress, while long-run performance tests are used to assess the performance of an application over an extended period of time.
Scalability tests are used to determine whether an application can handle an increase in workload if the configuration is modified.
Performance test When?
Performance testing should be conducted at appropriate stages of the development process to ensure the stability and performance of the complete system.
It is important to run unit performance tests for each component to ensure that the performance of the individual components does not negatively impact the overall system.
The full performance test of the complete application should be conducted as early as possible in the development process, to allow sufficient time to identify and fix any performance issues that may arise. Many performance issues can only be addressed through architecture modifications or deep code fixes, which can take time to implement.If performance testing is delayed until just before the production start date, there may not be sufficient time to address any issues that are identified.
This can lead to applications being patched for performance issues after they have been deployed, which can have a negative impact on the business, especially if the application is publicly accessible.
Knowing the application to test
Before conducting a performance test, it is essential to have a good understanding of the application being tested.
This includes identifying the various components of the application and their respective functions.
For example, the application may include components such as a runtime, a graphical user interface (GUI), a repository, or a router.
It is also important to determine the technologies being used in the application, such as databases and message systems.
An architecture diagram can be helpful in visualizing the components and technologies being used, and can serve as a reference point for discussing the performance targets and dimensions to be explored during the performance test.
Determine the performance goal
To properly evaluate the performance of an application, it is necessary to define specific performance goals for each important component of the application.
These goals should be non-functional, meaning that they relate to the performance of the system rather than its functional capabilities.
It is also important to consider the current and future service level agreements (SLAs) for the application.
The main performance goal should be the current SLA, but it is also important to determine whether the application can meet the future SLA.
For example, the performance goals for an application may include a requirement that the response time be less than 2 seconds and the throughput be at least 1000 transactions per second.
These goals should be clearly defined and used to guide the performance testing process.
Determine the benchmarks to run
A benchmark is a standard against which the performance of a system or component can be measured.
A benchmark is a set of tests that are designed to measure the performance of a system or component in a particular dimension, such as system responsiveness, memory usage, or data processing speed.
Your benchmarks must be repeatable.
The benchmark to run depends on the type of application.
A runtime application could be tested by a JMeter scenario.
A UI application could be tested by a selenium scenario or SPOT (https://github.com/IBM/spot-core/wiki)
Determine the dimension set to explore
The goal of the benchmarks is to examine and measure the performance of a system under certain conditions.
To do this, you need to select a dimension to focus on, such as:
-
- Horizontal scalability
- Vertical scalability
- Database size
- Impact of latency
- Payload size
- Responsiveness of the UI
- Complexity of the request in the application
- Number of concurrent executions
- Security level
- Encryption level
- Memory usage
- Role level
To explore a dimension, you will run multiple benchmarks while varying the chosen dimension.
For instance, if you want to test horizontal scalability, you would increase the number of servers and users in each benchmark run and observe how the system's performance changes.
Build a stable benchmark environment
To ensure that benchmark results are reliable and comparable, it is important to build a stable infrastructure for running the tests.
A stable infrastructure is one that can consistently produce similar performance results and does not encounter errors when running a benchmark multiple time.
It should also be readily available and equipped with the necessary tooling.
If it is not possible to obtain a stable infrastructure, or if there is a hardware upgrade, it may be necessary to use a tool to compare the theoretical performance of the different infrastructure setups in order to compare the results of two runs.
This will allow you to accurately assess the performance of the system under test.
Measure the baseline
To properly measure the performance of a system or application, it is important to compare the results of benchmark tests to a baseline.
The baseline represents the expected or normal performance of the system, and it can be used as a reference point to determine whether the system is performing as expected.
In order to establish a baseline, it is necessary to run the benchmarks on a stable system with a consistent configuration (no pod eviction...).
It is also important to record the configuration and methods used to run the benchmarks, so that the tests can be repeated if necessary.
It is generally recommended to validate the baseline by running the benchmarks multiple times, as a single run may not be sufficient to accurately measure performance.
It is also important to establish a separate baseline for each performance metric that is being measured.
Performance metrics to collect
Performance metrics are measures of the characteristics or qualities of a system that are used to evaluate its performance. These metrics can be used to measure the performance of a system in terms of various factors, such as speed, reliability, scalability, and resource usage.
Some common performance metrics for software systems include:
-
- Response time: This is the time it takes for a system to respond to a request.
- Throughput: This is the rate at which a system can process requests or perform a particular task.
- Resource utilization: This is the amount of resources (such as CPU, memory, or network bandwidth) that a system is using at a particular time.
- Error rate: This is the percentage of requests that result in an error or fail to be completed successfully.
- Availability: This is a measure of the percentage of time that a system is operational and able to process requests.
Performance metrics can be used to identify bottlenecks or other issues that may be impacting the performance of a system, and can help guide efforts to improve the system's performance.
If your system is complex, you need performance metrics from the different components.
For example, if your system is using a database, you need the CPU and network usage on yourapplication and on the database. Without this information it would be difficult to determinewhat is the next bottleneck to work on.
Performance tests cycle
You need to use this cycle per dimension that you want to explore.
Determine bottlenecks
If the system under test is not meeting the desired performance levels, it is necessary to identify the root cause of the issue.
This could be due to factors such as network configuration, insufficient CPU, slow disk, or inadequate memory.
Keep in mind that fixing one bottleneck may reveal another bottleneck, so you may need to iterate and address multiple bottlenecks.
If you are unable to identify the cause of the bottleneck, it may be necessary to review the list of performance metrics and consider whether the benchmarks are adequately measuring the relevant metrics and are well-defined.
This can help you understand and improve the system's performance.
Profiling the application or a component
Profiling an application can be a challenging task, as it requires using a tool to analyze the application's performance without altering its behavior.
The results of the profiling may indicate the presence of bottlenecks in the code or configuration that need to be addressed in order to improve the application's performance.
It is important to choose a profiling tool that has minimal impact on the application's performance and behavior, such as Instana, which is used in our case on OpenShift.
However, it is worth noting that such tools can have an impact on network bandwidth.
Refining tuning
Optimizing the configuration and performance of both the application and the underlying infrastructure can significantly impact the scalability and performance of your system.
This tuning can occur at various levels, including application tuning, open-source tuning, Kubernetes tuning, and HA-proxy tuning.
To effectively tune the system, it is necessary to have a deep understanding of the components of the application and how they can be optimized.
For example, if the system uses Elastic Search, you may need to adjust the number of shards and the allocated Java memory.
These tuning settings should become the default configuration for future baseline measurements and benchmark runs.
Rerun benchmarks
In order to understand the performance impact of a single modification to the system, it is important to rerun the benchmarks with only one change applied at a time.
This change could be a tuning of a single component or a modification to the configuration.
By rerunning the benchmarks with a single modification, you can evaluate the effect of that change on the system's performance.
It is crucial to carefully review the results of these tests in order to validate the tuning and decide on the next steps for performance optimization.
Determine the limits of the application and system
One of the objectives of performance testing is to identify the limitations of the application and system under test.
These limitations can be influenced by various factors such as the components of the application and the underlying infrastructure.
Therefore, optimizing these elements can have a significant impact on the limits of the system.
However, some limitations may be inherent to the application itself. In these cases, it may be necessary to consider creating a technical debt in order to address these limitations in a future release or build.
This will help to ensure that the application performs optimally.
Build a non-regression test
A performance non-regression test is a type of automated test that aims to ensure that the performance of the system has not degraded over time.
It typically covers a significant portion of the performance measures and is based on a subset of the performance benchmarks.
In order to minimize the number of false positives, it is important to run the non-regression test on a stable part of the application.
However, it can be challenging to find a stable infrastructure for running these tests, as it is necessary to ensure that any regressions detected are genuine and not the result of fluctuations in the infrastructure.
Conclusion
Performance testing is an ongoing process that should be conducted as early as possible and on a continuous basis.
It is important to run performance tests after every fix, even if the fix is related to security or another non-performance related issue, as the performance of the system may be impacted by these changes.
Non-regression tests can help to detect performance regressions quickly, allowing you to address any issues in a timely manner.
Overall, performance testing is essential for improving and optimizing the performance of your system, and it can help ensure that the system is using resources efficiently, which is a key consideration in sustainable or "green" practices.