AIOps

Objective

CloudPak for AIOps has an existing Instana integration which enables it to pull alerts, incidents, topology and metric data from Instana. Connection between the two products is fully configured via the wizard based integration, so it is the easiest in terms of setup.

This guide is an alternative approach to ingesting Instana alerts and incidents into AIOps. Instead of AIOps periodically polling for alerts/incidents you can configure a webhook endpoint in AIOps and then configure Instana to push the alerts/incidents and corresponding clears in real time. This is more efficient but does require additional configuration and exchange of certificates to fully configure.

Environment

At time of writing (October 2024) the environments used were:

RedHat Openshift v4.14

CloudPak for AIOps v4.7.0

Instana Standard Edition (k3s) v277

Prerequisites

Above environments deployed and Instana agents reporting data with alerts already being created in Instana.

High Level Steps

• Create AIOps Generic Webhook
• Create Instana Alert Channel
• Configure Alert Channel to be used to send Alerts to AIOps
• Add AIOps certificate (or Certificate Authority) to Instana truststore to create trust between the two products.

Steps

• Create AIOps Generic Webhook: From the CloudPak for AIOps console select Integrations->Add Integration and search for "webhook"

• Click the "Generic Webhook" tile shown above.
• Give the Webhook Integration a name and select your required Authentication type as below and click Next

• This next step is the key element of the webhook as you need to map an Instana alert to an AIOps event via JSONata.

Example JSONata (adjust as necessary for your needs): 

{
   "sender":{
      "service": issue.service,
      "name": "Instana",
      "type": "Webhook Connector"
   },
    "links": [
        {
        "url": issue.link
      }
    ],
   "resource":{
      "application": issue.application,
      "name": issue.entityLabel,
      "hostname": issue.hostname,
      "type": issue.entity,
      "ipaddress": issue.ipaddress,
      "location": issue.location
   },
   "type":{
      "classification": issue.metricNames[0],
      "eventType": $lowercase(issue.state) = "open" ? "problem": "resolution"
   },
   "severity": issue.severity = 10 ? 6 : issue.severity = 5 ? 3 : 1,
   "summary": issue.text,
   "occurrenceTime": issue.start
}

• Click Done.

A new pod will spin up in AIOps that begins with "ibm-grpc-webhook-connector...". Wait for the pod to spin up and you can then copy the "Webhook route" from AIOps to an Instana Alert Channel for use later in this procedure.

• Create Instana Alert Channel: From the Instana console select Settings-->Alert Channels-->Add Alert Channel-->Generic Webhook
• Enter a Name and paste the Webhook Route from the earlier step from AIOps. Click Create/Save

• Configure Alert Channel to be used to send Alerts to AIOps: From the Instana console select Settings-->Alerts-->New Alert
• Enter a Name. Select the types of events you want to be sent in section 2.
• Select the scope of the events to be sent in Section 3, e.g. Only on a subset of resources or Application perspectives. 
• Click "Add Alert Channels" in section 4 and select the new Alert Channel you created earlier.
• Click Create

• Add AIOps certificate (or Certificate Authority) to Instana: The AIOps webhook will be using your custom certificate or the cluster default certificate. This certificate is likely a self-signed certificate which Instana will not trust by default. The certificate (or CA) needs to be added to the Instana truststore to create trust between the two products. Only then will alerts be successfully sent and appear on the AIOps alert list.
  • Download the webhook certificate from the Openshift console or via an openssl command against the endpoint.
    • Example command: 
      • openssl s_client -showcerts -servername whconn-1a16589e-8e30-421e-a046-9ec7c4434a60-cp4aiops.apps.mysite.com -connect whconn-1a16589e-8e30-421e-a046-9ec7c4434a60-cp4aiops.apps.mysite.com:443 </dev/null
  • Save the certificate to a file and transfer to your Instana server.
  • Import the certificate to Instana. Instana documentation for importing a certificate to its truststore is here.
    • Example command: 
      • stanctl backend apply --core-custom-ca-crt <aiopsca.pem>

• Any active alerts in Instana should now successfully send to the AIOps webhook and appear in the AIOps alert list.