Co-Authors：
Hai Jun Xu (xuhjbj@cn.ibm.com) - Developer for Instana Synthetic PoP
Jun Shi Wang (wjunshi@cn.ibm.com) - Developer for Instana Synthetic PoP
Li Jing Mu (lijingmu@cn.ibm.com) - Developer for nstana Synthetic PoP
Rong Zhu Shang (shangrz@cn.ibm.com) - Developer for Instana Synthetic PoP
Rui Yin (yinrui@cn.ibm.com) - Tech lead for Instana Synthetic
What is Synthetic PoP?
Instana Synthetic monitoring, also known as proactive monitoring, can simulate actions that a user takes on an application from different locations, and continuously monitors at a specific interval for performance characteristics such as availability and response time. Synthetic monitoring is built on Instana, and it seamlessly integrates with other Instana capabilities. With the Synthetic monitoring features, you can create Synthetic tests to monitor an application.
Instana Synthetic PoP (Point of Presence) is the agent in which the Synthetic tests are running on.
Synthetic PoP is a cloud-native application that consists of Synthetic PoP controller, Redis, HTTP playback engine, JavaScript playback engine, and Browser playback engine.
Different Synthetic tests will trigger some synthetic PoP playback engines overloading. Therefore, there is a need to find an auto scaling solution to better manage the Synthetic PoP.
HPA and its limitations
Currently, Horizontal Pod Autoscaler (HPA) on Kubernetes only reacts to resource-based metrics such as CPU, memory usage, or custom metrics. However, using resource-based metrics has some limitations:
1)Too many pods are scaled up
Because different synthetic tests are dynamic consuming the resources, if HPA is based on the CPU or Memory usage, there will be a waste of resources.
For example, when HPA is used, there can be 5 browser tests running within 3 browser playback engine replicas. However, based on benchmark test results, there can be only 1 browser playback engine without HPA.
2) Pod flapping
Number of pod replicas keeps fluctuating frequently due to the dynamic nature of the metrics evaluated. e.g. stabilization window close to test frequency, Pod flapping due to fluctuating metrics in short time with number :1->3->2->3->2. ….
How to provide better scaling for Synthetic PoP based on customers’ Synthetic tests and available resources？
What is KEDA?
KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can scale any container in Kubernetes based on the number of events that need to be processed.
KEDA is proactive in its approach. It monitors the event source and provides this data as a custom metric to the HPA resource. This allows KEDA to scale containers based on the number of events that need to be processed before CPU or memory usage increases.
This blog introduces the basics of KEDA using a Redis trigger, and explains how to auto scale the Synthetic pods with Redis queue.
Auto Scaling Synthetic PoP with KEDA
Synthetic PoP deployment
The instructions for deploying Synthetic PoP can be found on Synthetic PoP Deployment.
Here is deploy the Synthetic PoP without TLS method:
After installing PoP, ensure that each pod is in the 'Running' and 'Ready' status.
Run the command: kubectl get pod -n synap
The result might be like the following:
Deploying KEDA
The instructions for deploying KEDA are implemented and can be found on KEDA's deployment page.
· What gets deployed in a KEDA deployment?
As the following example shows, the deployment contains the KEDA operator, roles and role bindings, as well as these custom resources:
- ScaledObject: The ScaledObject maps an event source to the deployment that you want to scale.
- TriggerAuthentication: If required, this resource contains the authentication configuration that is needed for monitoring the event source.
The scaled object controller is responsible for creating the HPA.
Event sources and scalers:
KEDA has a wide range of scalers that capable of both detecting whether a deployment should be activated or deactivated, and providing custom metrics for specific event sources. For example, Redis Lists | KEDA will be used by Synthetic PoP for scaling.
Redis Lists Trigger Specification:
Although KEDA supports multiple types of event sources, this article uses the Redis scaler. You can see the YAML for Redis Lists scaler as this below:
triggers:
- type: redis
metadata:
address: localhost:6379 # Format must be host:port
usernameFromEnv: REDIS_USERNAME # optional
passwordFromEnv: REDIS_PASSWORD
listName: mylist # Required
listLength: "5" # Required
activationListLength: "5" # optional
Define Custom Resources For Synthetic PoP
Now it is necessary to define ScaledObject and trigger authentication to implement. here are three playback engines that can be auto scaled based on the Redis Lists size. The following example focuses on the browser playback engine:
Define ScaledObject for browser playback engine
Find the secret name with synthetic-pop-redis for accessing the Redis pod in syn namespace:
root@testversion1:~# kubectl get secrets -n synap |grep synthetic-pop-redis
synthetic-pop-redis Opaque 1 20d
root@testversion1:~# kubectl edit secrets synthetic-pop-redis -n synap
Define the TriggerAuthentication named keda-trigger-auth-redis-secret based on above data setting of secret to access the Redis pod in the namespace:
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: keda-trigger-auth-redis-secret
namespace: synap
spec:
secretTargetRef:
- parameter: password
name: synthetic-pop-redis
key: redis-password
3.3. Define the Scaleobject yml file named browserscript-deployment-hpa.yaml for scalling browser script playback engine with Redis scaler, which includes the above TriggerAuthenticaton part:
kind: TriggerAuthentication
metadata:
name: keda-trigger-auth-redis-secret
namespace: synap
spec:
secretTargetRef:
- parameter: password
name: synthetic-pop-redis
key: redis-password
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: browserscript-scaledobject
namespace: synap
labels:
deploymentName: browserscript-scaledobject
spec:
minReplicaCount: 1
maxReplicaCount: 2
pollingInterval: 30
cooldownPeriod: 300
scaleTargetRef:
name: synthetic-pop-browserscript-playback-engine
advanced:
restoreToOriginalReplicaCount: true
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 100
periodSeconds: 15
triggers:
- type: redis
metadata:
address: synthetic-pop-redis.synap.svc.cluster.local:6378
listName: BrowserScriptPlaybackConfigQueue
listLength: "2"
activationListLength: "2"
enableTLS: "false"
unsafeSsl: "false"
databaseIndex: "0"
authenticationRef:
name: keda-trigger-auth-redis-secret
With the specifications described above, certain key parameters are needed to configure the browser playback engine:
· metadata:
name: browserscript-scaledobject, which is defined by the customer.
namespace: synap, which is the same as the Synthetic PoP used.
· scaleTargetRef:
name: synthetic-pop-browserscript-playback-engine, which is the deployment name of the browser script playback engine.
· triggers:
metadata.address: According to Redis list scaler, this should be the Redis host:port, with the Synthetic pop, that Redis service provides the access point as: synthetic-pop-redis.<namespace>.svc.cluster.local:6378. For example, we should use the synthetic-pop-redis.synap.svc.cluster.local:6378.
Metadata.listName: It is a reference to the BrowserScriptPlaybackConfigQueue , which is used for browser playback engine to get playback tasks.
metadata.listLenth:It triggers the scaler when above the value.
enableTLS: Allow a connection to a redis queue using tls. (Values: true, false, Default: false, Optional)
unsafeSsl: Used for skipping certificate check e.g: using self-signed certs. (Values: true,false, Default: false, Optional, This requires enableTLS: true).
· authenticationRef: It is a reference to the triggerAuthentication defined above with the name keda-trigger-auth-redis-secret.
Enable Synthetic PoP to support KEDA
Enable the KEDA operator to access the Redis service
As the Redis service define some network policy for security, so there is a need to update the network policy to allow the KEDA namespace to access the Redis service:
root@testversion1:~# kubectl edit networkpolicy synthetic-redis-ingress-network-policy -n synap
Add the below content and saved:
ingress:
- from:
- podSelector:
matchLabels:
component: synthetic-pop-controller
- podSelector:
matchLabels:
role: playbackengine
- namespaceSelector:
matchLabels:
name: keda
Notice : From release 261, customers can activate KEDA by simply configuring the keda.enabled and keda.namespace parameters in the synthetic installation helm commands.
Create the browserscript scaledobject
kubectl apply -f browserscript-deployment-hpa.yaml
Check the results
kubectl get scaledobject -n synap
NAME SCALETARGETKIND SCALETARGETNAME MIN MAX TRIGGERS AUTHENTICATION READY ACTIVE FALLBACK PAUSED AGE
browserscript-scaledobject apps/v1.Deployment synthetic-pop-browserscript-playback-engine 1 1 redis keda-trigger-auth-redis-secret True True False Unknown 65m
root@testversion1:~/keda# kubectl get hpa -n synap
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
keda-hpa-browserscript-scaledobject Deployment/synthetic-pop-browserscript-playback-engine 0/2 (avg) 1 2 1 68m
Verify the auto scaling
root@testversion1:~/keda# kubectl get pods -n synap
NAME READY STATUS RESTARTS AGE
synthetic-pop-redis-697d86f6fd-rr58h 1/1 Running 0 47h
synthetic-pop-controller-5cd88f497-6857r 1/1 Running 0 47h
synthetic-pop-browserscript-playback-engine-5644d5fc44-w2qpj 1/1 Running 0 46h
……
When there are more browser tests scheduled:
This will result in more than two sizes of Redis lists, and the ScaleObject will trigger the HPA to scale up the browser script playback engine pods.
root@testversion1:~/keda# kubectl get hpa -n synap
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
Deployment/synthetic-pop-browserscript-playback-engine 11/2 (avg) 1 2 2 84m
root@testversion1:~/keda# kubectl get pods -n synap
NAME READY STATUS RESTARTS AGE
synthetic-pop-redis-697d86f6fd-rr58h 1/1 Running 0 2d
synthetic-pop-controller-5cd88f497-6 1/1 Running 0 2d
synthetic-pop-browserscript-playback-engine-5644d5fc44-w2qpj 1/1 Running 0 46h
synthetic-pop-browserscript-playback-engine-5644d5fc44-b92qf 0/1 Pending 0 94s
……
After a while, that new browser script playback engine is running:
root@testversion1:~/keda# kubectl get pods -n synap
NAME READY STATUS RESTARTS AGE
synthetic-pop-redis-697d86f6fd-rr58h 1/1 Running 0 2d
synthetic-pop-controller-5cd88f497-6 1/1 Running 0 2d
synthetic-pop-browserscript-playback-engine-5644d5fc44-w2qpj 1/1 Running 0 46h
synthetic-pop-browserscript-playback-engine-5644d5fc44-b92qf 1/1 Pending 0 2m
……
Security Enhancement
Synthetic PoP support the Redis with TLS for security enhancement in production environment, details in this guide.
Since KEDA version 2.12,the Redis List Scaler support the TLS authentication. We can make below changes to achieve the enhancement with Synthetic PoP with Redis TLS.
Re-define TriggerAuthentication
Because from the guide, afterconfigure the Synthetic PoP with Redis TLS, there should be X.509 certificate-key pair (tls.crt, tls.key) and Certificate Authority (CA) root certificate file (ca.crt) in the secret named pop-tls-secret.
We can re-define the TriggerAuthentication named keda-trigger-auth-redis-secret based on above data setting of pop-tls-secret to access the Redis pod in the namespace:
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: keda-trigger-auth-redis-secret
namespace: synap
spec:
secretTargetRef:
- parameter: ca
name: pop-tls-secret
key: ca.crt
- parameter: cert
name: pop-tls-secret
key: tls.crt
- parameter: key
name: pop-tls-secret
key: tls.key
Re-define the Scaleobject yml to enable both TLS and unsafeSsl:
……………………………………………………
triggers:
- type: redis
metadata:
address: synthetic-pop-redis.synap.svc.cluster.local:6378
listName: BrowserScriptPlaybackConfigQueue
listLength: "2"
activationListLength: "2"
enableTLS: "true"
unsafeSsl: "true"
databaseIndex: "0"
authenticationRef:
name: keda-trigger-auth-redis-secret
Conclusion
Throughout the deployment of Synthetic PoP and KEDA, we have provided detailed the nece including defining custom resources, configuring triggers, and ensuring KEDA's access to Redis, among other considerations.
In summary, the auto scaling solution using KEDA enhances the flexibility and efficiency to Synthetic PoP. It enables resource adjustment based on actual event demands, avoiding resource wastage and frequent Pod fluctuations.
Next Steps
For the future, we recommend continuous optimization and monitoring of auto-scaling performance, adjusting as synthetic tests and resource situations evolve.
Furthermore, ongoing improvements to auto scaling strategies and KEDA configurations will help ensure the reliability of Synthetic PoP during periods of high load. A more detailed guide with be posted. With the methods outlined in the blog. we aim to assist you in effectively managing and auto scaling your Synthetic PoP for improved performance and availability.
If you require further assistance or have additional questions, reach out to the authors.