In part 1, I talked about the two primary approaches to performance testing; a high-risk approach and a low-risk approach. In this blog post, I would like to start addressing essential guidelines of performance testing. The first guideline to start with, is baselining discussed below.

Baselining

First, let us start with the definition of a baseline. A baseline is the set of configuration parameters for a given software solution stack version that produces a repeatable/predictable performance behavior. For a given software solution, there may be many baselines made during the performance testing and tuning process. The process of baselining is shown in Figure 1 below.

Figure 1: Performance baselining process

As the diagram in Figure 1 shows, to obtain the baseline, you must run a performance test which is the first step labeled with Bubble 1. As you run the performance test, you monitor the software solution that you are testing (Bubble 2). If any errors happen during the test, a decision must be made whether these errors are significant in a way that would impact the performance behavior (Bubble 3). If they are, these errors must be handled so that they don’t happen during performance testing (Bubble 4). Such error handling may require any number of things; software defect resolution, software configuration change, etc. Once the changes are made, performance testing is restarted (back to Bubble 1). If the errors are resolved or the errors do not impact the performance behavior, then the next step is to collect/store key performance metrics (Bubble 5). These metrics are specific to a given software solution. The software solution stack should be configured to provide such performance metrics before starting the test. Once, the performance test is complete, the initial baseline is made (Bubble 7). If this initial baseline meets the performance goals, then the performance testing can be declared complete for the kind of test case that is tested for performance (Bubble 9). If the initial baseline does not meet the performance goals, a change must happen before the process of baselining restarts. Note that the Bubble 6 decision becomes relevant when there is already a baseline. But, for the initial iteration of the baselining process, a baseline does not exist and the purpose is to create the initial baseline. The Bubble 6 decision is an effort to compare the current test results against a previous baseline. If the performance test results show a performance degradation compared to the baseline, one or more changes that went into the test may have to be backed out.

As mentioned above, the performance baseline is for a specific test case. The number of baselines can increase quickly for a given test case. So, imagine how many baselines you could have for all test cases for a given software solution. Each baseline should be associated with the following:

• A set of test case parameters
• A set of configuration parameters
• A combination of specific versions of the various software solution stack layers
• Set of key performance metrics

In the next post, I will continue the discussion on the performance testing guidelines.