Instana Best Practises from CP4BA Performance Teams

1.0     Use “zone” for multiple clusters

Go to “Infrastructure” view:

This is particularly useful for Instana to observe multiple clusters. As a result, your clusters details would not mix up. The screenshot above is an example by zoning to identify different clusters in infrastructure view.

Zoning is specified from instana agent yaml.

2.0      Add “Application” perspective to focus on specific details

Go to “Applications” view:

In Applications, “All services” is by default, which aggregate all details together. It becomes a problem when you have multiple clusters. All clusters’ details are merged in “All Services” view.
Create or add “Applications” in order to differentiate details between clusters in this case. Furthermore, you can add applications to monitor additional details in more depth.

3.0      Graphical view to identify services dependencies

Go to “Applications”, then choose “Dependencies”:

The example (as screenshot above) shows the interactions between services. The dots between services are in motion in the actual UI, which represents the actual traffic. In this example, bawdos service icon is enlarged and highlighted for latency issue. Jdbc and ldap in red alerts for service failure issue.

4.0      Enable alerts

Instana has extensive alert features in various areas. Since we talk about Applications, so we first show you how we setup alerts in application level. Then, the second section focuses on alerts for infrastructure and platforms based on either built in events or custom events.

4.1       Applications

Go to “Applications” view, then choose one of the applications:

Add Smart Alert is only available from application, and alerts can be triggered from erroneous calls, slow calls, unexpected high or low calls, and HTTP status codes. Performance team favorite alerts would be slow calls, and erroneous calls. One to catch performance issue and the other alert to ensure sound condition of the services.

4.2       Infrastructure & Platforms

Go to “Settings”, Select “Alerts”, then click “New Alert”:

Edit Alert > Select Alert on Event(s) > Add Events
In this example, under Entity type, choose RabbitMQ from the dropdown.
Performance engineers users alerts here to capture infrastructure & platform related noise or issues.

5.0      Use “Analytics” to investigate into issues

Go to “Analytics” view:

Instana has one of the most powerful analytics feature in the market. One of the favorite filters for performance team is “latency”. You can either use “call” or “trace” latency. In the example (as seen in the screenshot above), we use “call latency” to show any call longer than 3s. And because we have a few clusters link to this Instana, so we can specify the name to focus on one cluster only.

In additional to performance or latency issues, “Analytics” is also the view to troubleshoot your cluster. You can identify any erroneous calls.

The above screenshot is an example from SWAT investigating iam-scim issues using the Instana that kragon team provided.

6.0      Use “Time-Shift” feature to compare performance runs

Time Shift feature can be turned on from the dropdown as shown in the screen capture below:

Once Time Shift is set comparing against a specific time period, then the differences would be highlighted as the screen capture below:

7.0      Use “Instana AutoTrace Webhook”

Instana AutoTrace Webhook can enable Instana to auto trace Node.js, .Net Core, and Ruby application running across Kubernetes or OpenShift Cluster, IBM MQ, and ACE running in Cloud Pak.

In order to enable this feature in your cluster, additional steps are required.

Minimum requirement: OpenShift 4.5+, Helm 3.2+

Setup

helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook --repo https://agents.instana.io/helm instana-autotrace-webhook --set openshift.enabled=true --setautotrace.ingress_nginx.enable=true --set webhook.imagePullCredentials.password=<your Instana download key>

The sample command above applies to “OpenShift” and “Nginx and ingress-nginx”
NOTE: Please apply above to the api node in your cluster (where Instana agent was installed). Do not apply it to your Instana host.

Verify

kubectl get pods -n instana-autotrace-webhook
Execute above to check if the newly installed instana-autotrace-webhook pod is up & ready.

kubectl get pod test-nodejs -n test-apps -o=jsonpath='{.metadata.labels.instana-autotrace-applied}'
Execute above should return ‘true’.
Note: If you are having difficulty in getting ‘true, it is most likely you do not have a dummy ‘test-nodejs’ pod setup & running in namespace ‘test-apps’

kubectl logs -l app.kubernetes.io/name=instana-autotrace-webhook -n instana-autotrace-webhook
Execute above to get full logs of the instana-autotrace-webhook for further troubleshooting.

Upgrade

helm upgrade --namespace instana-autotrace-webhook instana-autotrace-webhook \
--repo https://agents.instana.io/helm instana-autotrace-webhook \
--reuse-values
Execute above to upgrade the webhook.

Remove

helm uninstall instana-autotrace-webhook --namespace instana-autotrace-webhook --no-hooks
Execute above to remove the webhook.

8.0      Customize Instana dashboard

Customize dashboard is not yet popularly used among performance teams. Most teams are using the default dashboards only. One main reason is most members are relatively new to Instana and exploring the built-in dashboards would take time to digest. Second reason is the feature is quite hidden. Although it is publicly advertised, not popularly adapted among the teams. The last reason, it is quite time consuming to edit widgets to suite our needs. More time needs to be devoted to just focus in a standard performance centric dashboard.

Go to Instana > Create Dashboard > Add Widget

Performance dashboard can be created as example below.

More time would be invested to create a dashboard that is designed to focus performance analysis that can be leverage among cp4ba performance teams.

9.0      Use Grafana as Instana front-end

Finding specific details in Instana, we may need to follow a sequence of steps. Instana has customizable dashboard feature, but Grafana UI, on the other hand, is more flexible and customizable. One of the popular ask among our customers in Instana tech talks is Grafana integration. Grafana dashboard features can also play a big role in performance engineering.

Below is an example of new dashboard with two graphs. First one shows ums latency in the past 30 days. And the second graph shows ADS runtime latency also in the last 30 days. As mentioned earlier, Grafana gives the flexibility of customization, so we get the benefit to grab what we need and compare the graphs side by side. 

Here is the summary of typical steps to setup:

·       Setup Grafana (docker version is recommended for its flexibility & lightweight)

·       Install instana plugin onto Grafana.

·       Add instana as a data source in Grafana (as shown below).

10.0      Backup your Instana stats

instanaDataBackup.sh

mkdir instanaDataBackup

instana stop

tar -cf ./instanaDataBackup/data.tar /mnt/data/

tar -cf ./instanaDataBackup/metrics.tar /mnt/metrics/

tar -cf ./instanaDataBackup/traces.tar /mnt/traces/

cp settings.hcl ./instanaDataBackup/

instana start