Co-Authors
Rong Zhu Shang(shangrz@cn.ibm.com) - Developer for Instana Synthetic PoP
Hai Jun Xu (xuhjbj@cn.ibm.com) - Developer for Instana Synthetic PoP
Rui Yin (yinrui@cn.ibm.com) - Tech lead for Instana Synthetic
About Instana Synthetic PoP
Synthetic PoP (Point of presence) is the agent where the synthetic tests are run. It consists of controller, Redis, HTTP playback engine, JavaScript playback engine, and Browser playback engine.
Performance Testing to Plan the PoP Configuration
Performance testing is done on a single PoP to determine the number of tests each playback engine can run. The tests results give you some directions on how to configure the Synthetic PoP based on your requirements. However, the test results are only for reference. The conditions in the production environment may largely vary. The number of tests is only indicative. You may increase or decrease the number of tests and PoP resources to achieve best results depending on your requirements. The capacity is different for playback engines. Synthetic PoP may become overloaded when you run more tests. In some cases, customers may want to reduce the resources of a PoP because of a small quantity of tests.
This blog post tries to help you with answers for the following questions:
- How to customize a PoP configuration?
- How to check whether a PoP is overloaded?
- How to troubleshoot when a PoP is overloaded?
Test Cases for Evaluating PoP Performance
The following configurations were defined for the test cases:
- HTTP Ping (or API simple) test: The response time is under 200 milliseconds, interval is 1 minute.
- API script test : The response time is under 800 milliseconds, interval is 1 minute. The test involves running 5 HTTP calls.
- Browser script test: The response time is under 20 seconds, interval is 5 minutes.
- Open http://www.ibm.com with one click to open another page and assert new page title.
Synthetic PoP requires different configuration of resources due to the number of Synthetic tests. If the number of tests is less, you can use small configuration. Similarly, you can use large configuration when number of tests is large. If there is not a test type, you can disable the playback engine.
The following playback engines are used for the tests:
- HTTP playback engine for API Simple test
- JavaScript playback engine for API Script test
- Browser playback engine for Browser test
To disable one of the playback engines, add the option to the helm command:
--set browserscript.enabled=false # disable browserscript engine
--set http.enabled=false # disable http engine
--set javascript.enabled=false # disable javascript engine
Monitor Synthetic PoP Using PoP Sensor
To check whether an engine is overloaded, you can use Synthetic PoP sensor. The sensor automatically checks Redis queue, scheduled tasks, and the test results to verify whether the pod is running normally. If a pod is overloaded, an overloaded event is triggered. You can see the event in the GUI. An overload event is shown in the following example.
Synthetic PoP configuration for medium configuration is covered in the following table.
|
Resource Allocation
|
PoP Controller
|
Redis
|
Http Engine
|
JavaScript engine
|
Browser script engine
|
Total
|
|
CPU Request
|
100m
|
100m
|
200m
|
600m
|
2000m
|
3000m
|
|
CPU Limit
|
150m
|
150m
|
300m
|
800m
|
4000m
|
5400m
|
|
Memory Request
|
200Mi
|
50Mi
|
200Mi
|
200Mi
|
2000Mi
|
2650Mi
|
|
Memory Limit
|
300Mi
|
100Mi
|
300Mi
|
300Mi
|
3000Mi
|
4000Mi
|
To install a Synthetic PoP with medium configuration, run this command:
helm install \
… \
--set browserscript.resources.limits.cpu=4000m \
--set browserscript.resources.limits.memory="3000Mi" \
--set browserscript.resources.requests.cpu=2000m \
--set browserscript.resources.requests.memory="2000Mi" \
--set controller.resources.limits.cpu=150m \
--set controller.resources.limits.memory="300Mi" \
--set controller.resources.requests.cpu=100m \
--set controller.resources.requests.memory="200Mi" \
--set http.resources.limits.cpu=300m \
--set http.resources.limits.memory="300Mi" \
--set http.resources.requests.cpu=200m \
--set http.resources.requests.memory="200Mi" \
--set javascript.resources.limits.cpu=800m \
--set javascript.resources.limits.memory="300Mi" \
--set javascript.resources.requests.cpu=600m \
--set javascript.resources.requests.memory="200Mi" \
--set redis.resources.limits.cpu="150m" \
--set redis.resources.limits.memory="100Mi" \
--set redis.resources.requests.cpu="100m" \
--set redis.resources.requests.memory=50Mi
3. Large configuration
The large configuration is used for running a higher number of tests.
|
Test type
|
Number of tests
|
Test frequency in minutes
|
Steps
|
|
HTTP
|
2000
|
1
|
1
|
|
API script
|
40
|
1
|
5
|
|
Browser script
|
10
|
5
|
2 *
|
*Two webpages are accessed as part of the test and the webpage results are verified.
Synthetic PoP configuration for the large configuration is covered in the following table.
|
Resource Allocation
|
PoP Controller
|
Redis
|
Http engine
|
JavaScript engine
|
Browser script engine
|
Total
|
|
CPU Request
|
100m
|
100m
|
200m
|
800m
|
3000m
|
4200m
|
|
CPU Limit
|
150m
|
150m
|
300m
|
1000m
|
5000m
|
6600m
|
|
Memory Request
|
200Mi
|
50Mi
|
200Mi
|
200Mi
|
3000Mi
|
3650Mi
|
|
Memory Limit
|
300Mi
|
100Mi
|
300Mi
|
300Mi
|
4000Mi
|
5000Mi
|
To install a Synthetic PoP with a large configuration, run the command:
helm install \
… \
--set browserscript.resources.limits.cpu=5000m \
--set browserscript.resources.limits.memory="4000Mi" \
--set browserscript.resources.requests.cpu=3000m \
--set browserscript.resources.requests.memory="3000Mi" \
--set controller.resources.limits.cpu=150m \
--set controller.resources.limits.memory="300Mi" \
--set controller.resources.requests.cpu=100m \
--set controller.resources.requests.memory="200Mi" \
--set http.resources.limits.cpu=300m \
--set http.resources.limits.memory="300Mi" \
--set http.resources.requests.cpu=200m \
--set http.resources.requests.memory="200Mi" \
--set javascript.resources.limits.cpu=1000m \
--set javascript.resources.limits.memory="300Mi" \
--set javascript.resources.requests.cpu=800m \
--set javascript.resources.requests.memory="200Mi" \
--set redis.resources.limits.cpu="150m" \
--set redis.resources.limits.memory="100Mi" \
--set redis.resources.requests.cpu="100m" \
--set redis.resources.requests.memory=50Mi
If a PoP has already installed, you can use helm upgrade command to only update the resource configuration without installing it again. The options for the helm upgrade and helm install commands are the same.
helm upgrade [options]
For vertical scaling, you need to configure CPU and memory. CPU/Memory for a single pod cannot be increased without limit. However, changing CPU and memory per playback engine is more flexible. For example, JavaScript playback engine is CPU intensive, so you can easily increase CPU instead of memory to support more API script workload.
Horizontal Scaling
Horizontal scaling requires increasing the Pods. Only one replica is available for each playback engine (HTTP, JavaScript, and Browser) by default. When one of playback engine is overloaded, you can scale up the engine replicas to handle more tests. The following example shows the default list of Synthetic Pods.
# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
synthetic-pop-browserscript-playback-engine 1/1 1 1 18d
synthetic-pop-controller 1/1 1 1 19d
synthetic-pop-http-playback-engine 1/1 1 1 19d
synthetic-pop-javascript-playback-engine 1/1 1 1 19d
synthetic-pop-redis 1/1 1 1 19d
In the example, all playback engines are ready. Only one playback engine is available by default.
Note: Only replica can be set for pop controller and Redis.
Scaling up JavaScript playback engine by using the Kubernetes command is shown in the following example.
Get the deployment of JavaScript playback engine, and then use kubectl scale deployment <deploy-name> --replicas=<num> command to update replicas.
# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
synthetic-pop-browserscript-playback-engine 1/1 1 1 18d
synthetic-pop-controller 1/1 1 1 19d
synthetic-pop-http-playback-engine 1/1 1 1 19d
synthetic-pop-javascript-playback-engine 1/1 1 1 19d
synthetic-pop-redis 1/1 1 1 19d
Set JavaScript playback engine replicas to 3.
# kubectl scale deployment synthetic-pop-javascript-playback-engine --replicas=3
deployment.apps/synthetic-pop-javascript-playback-engine scaled
Show pods list. The number Pods for JavaScript playback engine shown as 3.
# kubectl get pods
NAME READY STATUS RESTARTS AGE
synthetic-pop-browserscript-playback-engine-658d59d896-g5hnf 1/1 Running 0 18d
synthetic-pop-controller-5659c78cc4-nsjjt 1/1 Running 0 14d
synthetic-pop-http-playback-engine-c85c9d57f-jv5nz 1/1 Running 0 6d13h
synthetic-pop-javascript-playback-engine-6c5d488768-2s59c 1/1 Running 0 84s
synthetic-pop-javascript-playback-engine-6c5d488768-974cv 1/1 Running 0 42m
synthetic-pop-javascript-playback-engine-6c5d488768-zvj4m 1/1 Running 0 84s
synthetic-pop-redis-5cd4d547dc-prbs4 1/1 Running 0 19d
Horizontal scaling is easier to handle compared to vertical scaling. However, in some cases CPU and memory might be wasted. For example, JavaScript playback engine is CPU intensive, increasing the number of Pods for JavaScript playback engine may result in the wastage of memory.
Achieve Optimum Performance by Effective Use of Scaling
To achieve optimum performance for Synthetic PoP, use vertical scaling or horizontal scaling or use these methods in combination based your requirements.
You can make effective use of scaling, if you have a basic understanding of the resources required to support your workload and knowledge on how to scale up or scale down resources.
\