Stan's Corner

 View Only

How To Auto Scale Instana Synthetic PoP with KEDA

By Hai Jun Xu posted Thu October 26, 2023 01:10 AM

  

Co-Authors

Hai Jun Xu (xuhjbj@cn.ibm.com) - Developer for Instana Synthetic PoP

Jun Shi Wang (wjunshi@cn.ibm.com) - Developer for Instana Synthetic PoP

Li Jing Mu (lijingmu@cn.ibm.com) - Developer for nstana Synthetic PoP

Rong Zhu Shang (shangrz@cn.ibm.com) - Developer for Instana Synthetic PoP

Rui Yin (yinrui@cn.ibm.com) - Tech lead for Instana Synthetic

What is Synthetic PoP?

Instana Synthetic monitoring, also known as proactive monitoring, can simulate actions that a user takes on an application from different locations, and continuously monitors at a specific interval for performance characteristics such as availability and response time. Synthetic monitoring is built on Instana, and it seamlessly integrates  with other Instana capabilities. With the Synthetic monitoring features, you can create Synthetic tests to monitor an application.

Instana Synthetic PoP (Point of Presence) is the agent in which the Synthetic tests are running on.

Synthetic PoP is a cloud-native application that consists of Synthetic PoP controller, Redis, HTTP playback engine, JavaScript playback engine, and Browser playback engine.

 Different Synthetic tests will trigger some synthetic PoP playback engines overloading. Therefore, there is a need to find an auto scaling solution to better manage the Synthetic PoP.

HPA and its limitations

Currently, Horizontal Pod Autoscaler (HPA) on Kubernetes only reacts to resource-based metrics such as CPU, memory usage, or custom metrics. However, using resource-based metrics has some limitations:

1)Too many pods are scaled up 

Because different synthetic tests are dynamic consuming the resources, if HPA is based on the CPU or Memory usage, there will be a waste of resources.

For example,  when HPA is used, there can be 5 browser tests running within 3 browser playback engine replicas. However, based on benchmark test results, there can be only 1 browser playback engine without HPA.

2) Pod flapping

Number of pod replicas keeps fluctuating frequently due to the dynamic nature of the metrics evaluated. e.g. stabilization window close to test frequency, Pod flapping due to fluctuating metrics in short time with number :1->3->2->3->2. ….

How to provide better scaling for Synthetic PoP based on customers’ Synthetic tests and available resources

What is KEDA?


KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can scale any container in Kubernetes based on the number of events that need to be processed.

KEDA is proactive in its approach. It monitors the event source and provides this data as a custom metric to the HPA resource. This allows KEDA to scale containers based on the number of events that need to be processed before CPU or memory usage increases.

This blog introduces the basics of KEDA using a Redis trigger, and explains how to auto scale the Synthetic pods with Redis queue.

Auto Scaling Synthetic PoP with KEDA

Synthetic PoP deployment

The instructions for deploying Synthetic PoP can be found on Synthetic PoP Deployment.

Here is deploy the Synthetic PoP without TLS method:

After installing PoP, ensure that each pod is in the 'Running' and 'Ready' status.

Run the  command: kubectl get pod -n synap

The result might be like the following:

Deploying KEDA

The instructions for deploying KEDA are implemented and can be found on KEDA's deployment page.

·        What gets deployed in a KEDA deployment?

As the following example shows, the deployment contains the KEDA operator, roles and role bindings, as well as these custom resources:

  • ScaledObject: The ScaledObject maps an event source to the deployment that you want to scale.
  • TriggerAuthentication: If required, this resource contains the authentication configuration that is needed for monitoring the event source.

The scaled object controller is responsible for creating the HPA.

       Event sources and scalers:

KEDA has a wide range of scalers that capable of both detecting whether a deployment should be activated or deactivated, and providing custom metrics for specific event sources. For example, Redis Lists | KEDA will be used by Synthetic PoP for scaling.

Redis Lists Trigger Specification:

      Although KEDA supports multiple types of event sources, this article uses the Redis scaler. You can see the YAML for Redis Lists scaler as this below:

triggers:

- type: redis

  metadata:

    address: localhost:6379 # Format must be host:port

    usernameFromEnv: REDIS_USERNAME # optional

    passwordFromEnv: REDIS_PASSWORD

    listName: mylist # Required

    listLength: "5" # Required

    activationListLength: "5" # optional

Define Custom Resources For Synthetic PoP

Now it is necessary to define ScaledObject and trigger authentication to implement. here are three playback engines that can be auto scaled based on the Redis Lists size. The following example focuses on the browser playback engine:

Define ScaledObject for browser playback engine

Find the secret name with synthetic-pop-redis for accessing the Redis pod in syn namespace:

root@testversion1:~# kubectl get secrets -n synap |grep synthetic-pop-redis

synthetic-pop-redis                    Opaque                                1      20d

root@testversion1:~# kubectl edit secrets synthetic-pop-redis -n synap

Define the TriggerAuthentication named keda-trigger-auth-redis-secret based on above data setting of secret to access the Redis pod in the namespace:

apiVersion: keda.sh/v1alpha1

kind: TriggerAuthentication

metadata:

  name: keda-trigger-auth-redis-secret

  namespace: synap

spec:

  secretTargetRef:

  - parameter: password

    name: synthetic-pop-redis

    key: redis-password

3.3. Define the Scaleobject yml file named browserscript-deployment-hpa.yaml for scalling browser script playback engine with Redis scaler, which includes the above TriggerAuthenticaton part:

kind: TriggerAuthentication

metadata:

  name: keda-trigger-auth-redis-secret

  namespace: synap

spec:

  secretTargetRef:

  - parameter: password

    name: synthetic-pop-redis

    key: redis-password

---

apiVersion: keda.sh/v1alpha1

kind: ScaledObject

metadata:

  name: browserscript-scaledobject

  namespace: synap

  labels:

    deploymentName: browserscript-scaledobject

spec:

  minReplicaCount: 1

  maxReplicaCount: 2

  pollingInterval: 30  

  cooldownPeriod:  300

  scaleTargetRef:

    name: synthetic-pop-browserscript-playback-engine

  advanced:

    restoreToOriginalReplicaCount: true

    horizontalPodAutoscalerConfig:

      behavior:

        scaleDown:

          stabilizationWindowSeconds: 300

          policies:

          - type: Percent

            value: 100

            periodSeconds: 15

  triggers:

  - type: redis

    metadata:

      address: synthetic-pop-redis.synap.svc.cluster.local:6378

      listName: BrowserScriptPlaybackConfigQueue

      listLength: "2"

      activationListLength: "2"

      enableTLS: "false"

      unsafeSsl: "false"

      databaseIndex: "0"

    authenticationRef:

      name: keda-trigger-auth-redis-secret

With the specifications described above, certain key parameters are needed to configure the browser playback engine:

·       metadata:

name: browserscript-scaledobject, which is defined by the customer.

namespace: synap, which is the same as the Synthetic PoP used.

·       scaleTargetRef:

name: synthetic-pop-browserscript-playback-engine, which is the deployment name of the browser script playback engine.

·       triggers:

metadata.address: According to Redis list scaler, this should be the Redis host:port, with the Synthetic pop, that Redis service provides the access point as: synthetic-pop-redis.<namespace>.svc.cluster.local:6378. For example, we should use the synthetic-pop-redis.synap.svc.cluster.local:6378.

Metadata.listName: It is a reference to the BrowserScriptPlaybackConfigQueue , which is used for browser playback engine to get playback tasks.

metadata.listLenth:It triggers the scaler when above the value.

enableTLS: Allow a connection to a redis queue using tls. (Values: true, false, Default: false, Optional)

unsafeSsl: Used for skipping certificate check e.g: using self-signed certs. (Values: true,false, Default: false, Optional, This requires enableTLS: true).

·       authenticationRef: It is a reference to the triggerAuthentication defined above with the name keda-trigger-auth-redis-secret.

Enable Synthetic PoP to support KEDA

Enable the KEDA operator to access the Redis service

As the Redis service define some network policy for security, so there is a need to update the network policy to allow the KEDA namespace to access the Redis service:

root@testversion1:~# kubectl edit networkpolicy synthetic-redis-ingress-network-policy -n synap

Add the below content and saved:

  ingress:

    - from:

      - podSelector:

          matchLabels:

            component: synthetic-pop-controller

      - podSelector:

          matchLabels:

            role: playbackengine

      - namespaceSelector:

          matchLabels:

            name: keda

Notice : From  release 261, customers can activate KEDA by simply configuring the keda.enabled and keda.namespace parameters in the synthetic installation helm commands.

Create the browserscript scaledobject

kubectl apply -f browserscript-deployment-hpa.yaml

Check the results

kubectl get scaledobject -n synap

NAME     SCALETARGETKIND  SCALETARGETNAME   MIN   MAX  TRIGGERS AUTHENTICATION                   READY   ACTIVE   FALLBACK   PAUSED    AGE

browserscript-scaledobject  apps/v1.Deployment   synthetic-pop-browserscript-playback-engine   1     1     redis      keda-trigger-auth-redis-secret   True    True     False      Unknown   65m

root@testversion1:~/keda# kubectl get hpa -n synap

NAME      REFERENCE   TARGETS     MINPODS   MAXPODS   REPLICAS   AGE

keda-hpa-browserscript-scaledobject   Deployment/synthetic-pop-browserscript-playback-engine   0/2 (avg)          1         2         1          68m

Verify the auto scaling

root@testversion1:~/keda# kubectl get pods -n synap

NAME                                                           READY   STATUS    RESTARTS   AGE

synthetic-pop-redis-697d86f6fd-rr58h                           1/1     Running   0          47h

synthetic-pop-controller-5cd88f497-6857r                       1/1     Running   0          47h

synthetic-pop-browserscript-playback-engine-5644d5fc44-w2qpj   1/1     Running   0          46h

……

When there are more browser tests scheduled:

This will result in more than two sizes of Redis lists, and the ScaleObject will trigger the HPA to scale up the browser script playback engine pods.

root@testversion1:~/keda# kubectl get hpa -n synap                                                                                                                                                 

NAME REFERENCE     TARGETS                    MINPODS   MAXPODS   REPLICAS   AGE                                       

Deployment/synthetic-pop-browserscript-playback-engine   11/2 (avg)  1 2 2  84m

root@testversion1:~/keda# kubectl get pods -n synap                                                                                                                                                 

NAME                 READY   STATUS    RESTARTS   AGE                                                                                                     

synthetic-pop-redis-697d86f6fd-rr58h        1/1     Running   0           2d                                                                                                     

synthetic-pop-controller-5cd88f497-6        1/1     Running   0           2d                                                                                     

synthetic-pop-browserscript-playback-engine-5644d5fc44-w2qpj 1/1 Running 0   46h                                                                                                     

synthetic-pop-browserscript-playback-engine-5644d5fc44-b92qf 0/1 Pending 0   94s

……

After a while, that new browser script playback engine is running:

root@testversion1:~/keda# kubectl get pods -n synap                                                                                                                                                

NAME                 READY   STATUS    RESTARTS   AGE                                                                                                     

synthetic-pop-redis-697d86f6fd-rr58h        1/1     Running   0           2d                                                                                                     

synthetic-pop-controller-5cd88f497-6        1/1     Running   0           2d                                                                                     

synthetic-pop-browserscript-playback-engine-5644d5fc44-w2qpj 1/1 Running 0   46h                                                                                                     

synthetic-pop-browserscript-playback-engine-5644d5fc44-b92qf 1/1 Pending 0   2m

……

Security Enhancement

Synthetic PoP support the Redis with TLS for security enhancement in production environment, details in this guide.

Since KEDA version 2.12,the Redis List Scaler support the TLS authentication. We can make below changes to achieve the enhancement with Synthetic PoP with Redis TLS.

Re-define TriggerAuthentication

Because from the guide, afterconfigure the Synthetic PoP with Redis TLS, there should be X.509 certificate-key pair (tls.crt, tls.key) and Certificate Authority (CA) root certificate file (ca.crt) in the secret named pop-tls-secret.

We can re-define the TriggerAuthentication named keda-trigger-auth-redis-secret based on above data setting of pop-tls-secret to access the Redis pod in the namespace:

apiVersion: keda.sh/v1alpha1

kind: TriggerAuthentication

metadata:

  name: keda-trigger-auth-redis-secret

  namespace: synap

spec:

  secretTargetRef:

  - parameter: ca

    name: pop-tls-secret

key: ca.crt

  - parameter: cert

    name: pop-tls-secret

key: tls.crt

  - parameter: key

    name: pop-tls-secret

key: tls.key

Re-define the Scaleobject yml to enable both TLS and unsafeSsl:

……………………………………………………

triggers:

  - type: redis

    metadata:

      address: synthetic-pop-redis.synap.svc.cluster.local:6378

      listName: BrowserScriptPlaybackConfigQueue

      listLength: "2"

      activationListLength: "2"

      enableTLS: "true"

      unsafeSsl: "true"

      databaseIndex: "0"

    authenticationRef:

      name: keda-trigger-auth-redis-secret

Conclusion

Throughout the deployment of Synthetic PoP and KEDA, we have provided detailed the nece including defining custom resources, configuring triggers, and ensuring KEDA's access to Redis, among other considerations.

In summary, the auto scaling solution using KEDA enhances the flexibility and efficiency to Synthetic PoP. It enables resource adjustment based on actual event demands, avoiding resource wastage and frequent Pod fluctuations.

Next Steps

For the future, we recommend continuous optimization and monitoring of auto-scaling performance, adjusting as synthetic tests and resource situations evolve.

Furthermore, ongoing improvements to auto scaling strategies and KEDA configurations will help ensure the reliability of Synthetic PoP during periods of high load. A more detailed guide with be posted. With the methods outlined in the blog. we aim to assist you in effectively managing and auto scaling your Synthetic PoP for improved performance and availability.

If you require further assistance or have additional questions,  reach out to the authors.

Permalink