IT Automation

 View Only

On generating alerts using the Webhook integration to test the CP4AIOps policy, a tutorial.

By JULIUS WAHIDIN posted 5 days ago

  

Recently, a customer asked me, "In CP4AIOps, How do I generate alerts to test the automation policy I am creating?". While responding to the question, I thought the answer might also be helpful to others, hence this blog.

This tutorial will go through the process step-by-step using the following plan:

  • First, we will deploy and configure the CP4AIOps (Cloud Pak for AI Operation) generic webhook. Many blogs exist, but not many go through the actual development of alert mapping.
  • Second, we will use one out-of-the-box sample configuration and send a test event using CURL.  Some troubleshooting steps are included in case you have issues following this tutorial.
  • Since the original purpose of generating the alerts is to test a policy, let's create a simple policy. We will configure a policy to produce a scope-based group.
  • Next, we will modify the webhook JSONata to customise the payload; you most likely want a specific field, including some custom fields, as input for your policy.
  • Generating JSON manually can be time-consuming. We need a way to create lots of alerts to test the policy quickly. The question at the beginning of this blog was asked to help the customers' business analysts test some policies. Most business analysts use spreadsheets, so we will create our alerts in a spreadsheet and save them as a CSV file.
  • Finally, we will use a simple Python script to ingest the CSV file and test our automation policy.

The screenshots in this tutorial were taken on CP4AIOps version 4.8.x, the most recent version at the time of writing.

If you want to follow along on your environment, you will need the following:

  • Access to CP4AIOps version 4.x.x (yes, it does not have to be the latest version) with a user role to edit policy and integration.
  • Access to a workstation with CURL and Python that can access the CP4AIOps UI.
ideas
If you have access to a traditional Netcool Omnibus system, another way is to inject the test alerts into Omnibus and let it replicate to CP4AIOps using the Netcool connector.   I have written a blog to use the Omnibus REST interface here.

Deploy a webhook integration.

Last year, I delivered training on CP4AIops. The students sent events to a generic webhook as part of the lab exercise. Upon seeing the alerts in the CP4AIOps console, one student had a lightbulb moment and shouted, “Aha! That was easy. Remembering the contagious positive response, let us use a webhook probe to send our test alerts. 

But first, we need to configure our webhook integration. Yes, CP4AIOps supports multiple webhook probes running simultaneously, so we can create our webhook.

Follow the following steps:

  1. Login to your AIOps system.
  2. Select Integration from the menu.
  3. Click the Add Integration blue button. If the Generic Webhook has been configured for your environment, you should click the Generic Webhook instead of steps 3 to 6 and then click Add Integration.
  4. Type in “webhook” in the search bar.
    Add Integration
  5. Select the Generic Webhook card.
  6. Click the Get Started option button.
  7. Name the webhook integration instances. Let us use our name; make this unique and personal, as only we will use this integration. 
  8. Specify a username and password. Paste this into a temporary text editor. We will need this to call the webhook.
  9. Leave the generated webhook route as is.
    WebhookIntegration
  10. Click Next. The fun part comes: We will map the incoming fields to the target alert fields.
  11. Some predefined mappings are available; click the Load sample mapping to select one.
  12. Let us select one with a single-layer JSON input. Simple is good. So choose the sevOne sample.
    Load Sample Mapping
  13. Study the generated JSONata configuration. We will not modify this JSONata yet; we want to test it first to ensure that our Generic probe runs OK—one step at a time.
  14. Click save. Wait about 2 minutes until the Generic Webhook integration status changes to a green check mark and is Running.
  15. When we click save, CP4AIOps spin a Generic Webhook pod; we will query this pod later during troubleshooting.
  16. Copy the webhook route of the integration we just created using the blue copy-to-clipboard icon. This route is the URL to send your alerts' JSON payload. We will need it later, so paste it into our temporary text editor.
    WebhookRoute
  17. The webhook route should look like this:  https://whconn-7c8dc065-839b-4ae8-99cd-1f9cd7185ee8-cp4aiops.apps.o2-182300.cp.fyre.ibm.com/webhook-connector/v91vf0g43gn.
  18. The webhook integration is now ready. 
  19. In the next section, we will send a JSON payload. Before that, let's understand a bit more about the sample JSONata mapping.

JSONata

The following is the JSONata from the sevOne sample and some comments to help understand it:

(
    {
        "severity": alertState="Alert" ? 6 : alertState="Emergency" ? 6 : alertState="Critical" ? 6 : alertState="Error" ? 5 : alertState="Warning"? 3 : alertState="Notice" ? 2 : alertState="Info"? 2 : alertState="Debug" ? 2 : 1,
        "summary": description,
        "resource": {
            "name": deviceName
        },
        "type": {
            "classification": policyName,
            "eventType": ($exists(closureMessage) and closureMessage != "N/A") ? "resolution" : alertState = "Cleared" ? "resolution" : "problem"
        },
        "sender": {
            "name": "IBM SevOne",
            "type": "Webhook Connector"
        },
        "expirySeconds": alertId="-1" ? 300
    }
)
  • JSONata defines a mapping from the input field on the right of the operand “:” to the output fields on the left. 
  • The format for every row is “output”: input. 
  • You can put a conditional statement on the input using the “condition ? true_value : false_value“.
  • If you look at the last condition of the row that assigns the severity, it displays '... alertState=” Debug”? 2: 1'. This means the output field “severity” will be assigned the integer 2 if the alertState field from the input sevOne JSON has the string value "Debug". Otherwise, it will be assigned the integer 1. The rest of the row assignment is a cascading condition statement.
  • So, the whole JSONata structure assumes the following single-level JSON input fields:
    • alertState
    • description
    • deviceName
    • policyName
    • closureMessage
    • alertId
  • We will simulate a sevOne sending a JSON record with the above fields, and it will be assigned to the following CP4AIOps alerts fields:
    • severity
    • summary
    • resource.name
    • type.classification
    • type.eventType
    • sender.name
    • sender.type
  • We can verify these fields in the generated alert.

Lightbulb

There are many possible sources of events feeding into CP4AIOps as alerts.  To name a few:

  • From external Omnibus probes.
  • From external Omnibus through Netcool connector.
  • Generated as synthetic events through Impact connector.
  • Through CP4AIOps alerts API.
  • Through Kafka (depending on CP4AIOps version).
  • Generated from CP4AIOps topology as resources status.
  • From external sources through CP4AIOps webhook integration.
This tutorial uses the last method, which generates the contagious positive response :D.

Test alerts

We will be sending a test alert using CURL.  To do that, we will need the following information:

  1. The Webhook Endpoint URL.  We already have this from the Webhook route earlier.
  2. Username and password. We will use basic authentication by decoding them in the CURL header.
  3. The Actual JSON payload. We will store the payload in files because you can edit and reuse files. This is a better alternative to specifying the payload on the command line.

Authorization

We will use basic authentication by encoding our username and password using base64 by performing the following (I am using a standard RHEL 9.x workstation).

$ echo -ne "julius:WhatIsAGoodPasswordFor2025?" | base64 --wrap 0 ; echo 
anVsaXVzOldoYXRJc0FHb29kUGFzc3dvcmRGb3IyMDI1Pw==

Note the use of -n in the first echo command; this removes any trailing newline character. The wrap 0 in the base64 command serves the same purpose. The last echo command makes the output more readable and makes it easier to copy the generated encoded value.

The resulting decoded string may not produce the desired result without these options.

JSON payload

We have purposely selected the sevOne sample webhook mapping, as it uses a flat JSON structure, which is easier to work with.  Let us create a file named sevOneEvent.json containing one sevOne alert:

$ cat sevOneEvent.json 
{
  "alertState": "Alert",
  "description": "Sample description",
  "deviceName": "hostA",
  "policyName": "sample",
  "closureMessage": "N/A",
  "alertId": "-1"
}

Sending the event.

We now have everything to send the test event. Manually entering and correcting any long command can be challenging.  Putting the CURL command in a file and running it is more manageable for us as human beings (unless you are an Android).

$ cat sendSevOneEvent.sh 
curl -k -X POST \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Basic anVsaXVzOldoYXRJc0FHb29kUGFzc3dvcmRGb3IyMDI1Pw==' \
  https://whconn-7c8dc065-839b-4ae8-99cd-1f9cd7185ee8-cp4aiops.apps.o2-182300.cp.fyre.ibm.com/webhook-connector/v91vf0g43gn \
  -d "@sevOneEvent.json"

$ ./sendSevOneEvent.sh 
{"Status":200,"UID":"cc5693fb-a4bd-415c-ad6a-6cd2bf1c6084","Message":"Event received."}

Note the Status 200 status response, which tells us the command succeeded. If you do not see this, read the troubleshooting guide below.

If we check our alerts in CP4AIOps, we should see the event we just sent.

sevOneAlert

Woohoo!  Our webhook integration works.  It's time for some small dance—tra la la la la.

Troubleshooting

If your command does not work, then there are a few steps that you can do.

  1. Run the curl command verbosely by adding the -v option to the curl command.
  2. Check the log of the webhook integration pod.

The following are examples of errors as you run the CURL command.

{"Status":400,"ErrorMessage":"Invalid JSON payload or event failed validation check because it is missing mandatory event fields after JSONata evaluation."}

{"Status":406,"ErrorMessage":"Event mapping expression did not result in an output event object."}

Here are sample steps to check the log of the webhook integration pod.

  • First, get the pod name.  We are searching for a pod with "webhook" in its name.  I am selecting the youngest pod because I have two webhook integrations, and I just created the webhook pod for this tutorial.
oc get pods | grep webhook
ibm-grpc-webhook-connector-1ca30fd9-d836-45c0-8693-053e3bd5gjj8   1/1     Running     0              36d22h
ibm-grpc-webhook-connector-7c8dc065-839b-4ae8-99cd-1f9cd71qwbwr   1/1     Running     0               1h08m
  • Now, get the pod's logs. You might want to use the tail options to reduce the output to the last few lines of the log.
$ oc logs ibm-grpc-webhook-connector-7c8dc065-839b-4ae8-99cd-1f9cd71qwbwr --tail 10
Defaulted container "ibm-grpc-webhook-connector" out of: ibm-grpc-webhook-connector, cert-setup (init)
[2/18/25, 12:48:13:926 UTC] 0000003d ConnectorCred I   client id and secret provided
[2/18/25, 12:48:40:442 UTC] 00000042 WebhookConnec I   Authenticated, event received (156 bytes): {  "alertState": "Alert",  "description": "Sample description",  "deviceName": "hostA",  "policyName": "sample",  "closureMessage": "N/A",  "alertId": "-1"}
[2/18/25, 12:48:40:491 UTC] 00000042 WebhookValida I   Appended current time as occurenceTime in event.
[2/18/25, 12:48:40:492 UTC] 00000042 WebhookValida I   Processed event: {"occurrenceTime":"2025-02-18T12:48:40Z","summary":"Sample description","severity":6,"type":{"eventType":"problem","classification":"sample","condition":null},"sender":{"name":"IBM SevOne","type":"Webhook Connector"},"resource":{"name":"hostA"},"expirySeconds":300}
[2/18/25, 12:48:40:501 UTC] 00000042 WebhookConnec I   HTTP response code: 200, body: {"Status":200,"UID":"996b781b-eac9-4a9c-92fe-95afa56da444","Message":"Event received."}, processing time: 0.059 seconds

The above output shows a successful one. A log output with errors might look like the following (depending, of course, on the error):

[2/12/25, 9:26:27:118 UTC] 00000051 WebhookConnec I   Event received (184 bytes): {  "Severity": 6,  "Summary": "Sample Critical alert - Temperature too hot",  "Node": "hostA",  "AlertGroup": "Temperature",  "EventType": 1,  "AlertId": "-1",  "Scope": "Environment"}
[2/12/25, 9:26:27:151 UTC] 00000051 WebhookValida W   Parsing exception: line 3:20 at null: token recognition error at: '@'
, line 3:41 at null: token recognition error at: '@'  
, line 3:63 at null: token recognition error at: '@'  
, line 3:85 at null: token recognition error at: '@'  

[2/12/25, 9:26:27:164 UTC] 00000051 WebhookConnec W   Unable to process JSON event: Unrecognized field "Severity" (class com.ibm.cp4waiops.connectors.sdk.EventLifeCycleEvent), not marked as ignorable (10 known properties: "details", "sender", "summary", "links", "type", "id", "expirySeconds", "severity", "occurrenceTime", "resource"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 17] (through reference chain: java.lang.Object[0]->com.ibm.cp4waiops.connectors.sdk.EventLifeCycleEvent["Severity"])
[2/12/25, 9:26:27:172 UTC] 00000051 WebhookConnec I   HTTP response code: 406, body: {"Status":406,"ErrorMessage":"Event mapping expression did not result in an output event object."}, processing time: 0.055 seconds
[2/12/25, 9:28:40:528 UTC] 0000003f ConnectorMana I   writeConnectorStatus() Status: name=Running retry=0

The policy

We have tested the webhook integration, and it works. We can now proceed with customizing the JSON payload. 

Since the main problem we tried to address in this tutorial is to inject alerts to test a policy, we need to know the alert fields required by the policy. 

In this tutorial, we want to test a simple policy; a scope-based grouping policy is simple enough. A Scope-based policy will need at least one field as the scope. So, let us create the policy.  Following along the steps below:

  • From the main menu, select Automations.
  • Click the Create policy blue button.
  • Click the Create word on the Group alerts based on scope.
  • Enter the policy name and optional description in the Policy details section.
  • You can enter any number for the execution order. I entered 88 to represent that our policy is relatively the least important; besides, in some cultures, 88 is a lucky number :D.
    Policy Details
  • The following section configures the policy triggers and specifies when the policy should be run. We select both the Before an alert is created and the After an alert has been updated when the lastOccurrenceTime changes. This means this policy is run when CP4AIOps detects new and existing alerts that had the lastOccurrenceTime updated since the last time the check was done. 
Policy Trigger

Lightbulb
Another alert field commonly used to verify the updated alerts is Severity. If we specify a change in the Severity policy trigger, we can manually run the policy by changing the alert's severity through the UI.

  • Even though the policy is now running, we can specify one or more conditions that will act as a gatekeeper and perform a test on the alert's payload. We can specify zero or more conditions. It is recommended that we have at least one condition. For the condition here, we are only interested in processing the event from our webhook, so we are testing that the alert sender's name is Julius Webhook.   When we map our webhook integration later, we need to ensure that we assign this string.
    Policy Condition
  • Now comes the policy's action. The policy's purpose is to group alerts together based on a scope.  We need to specify the scope. For this tutorial, let us choose the scope based on the alerts' custom field: details.scope. Select the details field, then type in the subfield custom as shown, and then click apply.  We are configuring CP4AIOps to search for alerts with the same alert.details.scope content, and group them together. Remember, like the sender name, we will need to include the details.scope in our We will keep our webhook integration.
    policyAction
  • We will not modify the Time Window section, so go back to the top of the User Interface interaction, ensure the enable switch is set to enable and click Save.  Our policy is now ready.

Alerts mapping take 2

Earlier, we sent alerts manually through CURL with the sevOne payloads.  Now, we are creating our payload.  There are two compulsory fields expected by the policy we want to test: sender.name = "Julius Webhook", and there should be at least two alerts with the same details.scope.  We will put the sender.name directly as a literal string in our webhook mapping, so we only need to specify the same scope in our alerts payload. Let us create two files with the required JSON payload as follows:

alerts1.json file content
{
  "Node": "computer A",
  "Summary": "Hardisk 95% full",
  "Severity": 4,
  "AlertGroup": "Computer",
  "Scope": "ComputerSupport",
  "Service": "Support",
  "Location": "Sydney",
  "URL": "https://www.ibm.com",
  "URLdesc": "link"
}
alerts2.json file content
{
  "Node": "computer B",
  "Summary": "Hardisk 1 failed.",
  "Severity": 6,
  "AlertGroup": "Computer",
  "Scope": "ComputerSupport",
  "Service": "Support",
  "Location": "Melbourne",
  "URL": "https://www.ibm.com",
  "URLdesc": "link"
}

With the new payload, we need to modify our webhook mapping accordingly.  Here is the new webhook mapping.

(
    {
        "severity": $number(Severity) < 1 ? 1 : ($number(Severity) < 6 ? $number(Severity) : $number(Severity) >= 6 ? 6),
        "summary": Summary,
        "resource": {
            "name": Node,
            "service": Service,
            "location": Location
        },
        "links": {
            "linkType":"webpage",
            "name": URLdesc,
            "description": URLdesc,
            "url": URL
        },
        "type": {
            "classification": AlertGroup,
            "eventType": "problem"
        },
        "sender": {
            "name": "Julius Webhook",
            "type": "Webhook integration"
        },
        "expirySeconds": 300,
        "details": {
            "custom": Scope
        }
    }
)

Now, let us send the content of the files through CURL as before.

$ curl -k -X POST \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Basic anVsaXVzOldoYXRJc0FHb29kUGFzc3dvcmRGb3IyMDI1Pw==' \
  https://whconn-7c8dc065-839b-4ae8-99cd-1f9cd7185ee8-cp4aiops.apps.o2-182300.cp.fyre.ibm.com/webhook-connector/v91vf0g43gn \
  -d "@alerts1.json"
$ curl -k -X POST \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Basic anVsaXVzOldoYXRJc0FHb29kUGFzc3dvcmRGb3IyMDI1Pw==' \
  https://whconn-7c8dc065-839b-4ae8-99cd-1f9cd7185ee8-cp4aiops.apps.o2-182300.cp.fyre.ibm.com/webhook-connector/v91vf0g43gn \
  -d "@alerts2.json"
 

And here is what the alerts screen should look like.

GroupedAlerts

The alerts are grouped, and the correlation column shows they are scope-grouped. Yay, whoop whoop whoop.... 

lightbulb
Note that we have hardcoded the expirySeconds of 300 in the webhook mapping. The purpose of our alerts is to test a policy, not to reside in the existing alerts forever. As a recommended practice, always think of the alert lifecycle clean-up phase. We do not want a bloated system, hence auto-cleaning your alerts. 

More alerts, and then more...

We now need a way to let the business analysis specify more alerts. The easiest way to do this will be to use a spreadsheet and then save the content into a CSV.  Luckily, Python provides libraries that can read the content of a CSV file, convert it to JSON, and send it through HTTPS.

So, let us use a spreadsheet, and very quickly, we can create a sample of 10 alerts with the three overall scopes. In the real environment, we may want to study the actual alert of interest. We may want to replay those alerts by copying them into the spreadsheets. After all, we create and test our policy to be applied to real alerts.

spreadsheet

Here is the spreadsheet's CSV content so that you can test it in your environment.

Node,Summary,Severity,AlertGroup,Scope,Service,Location,URL,URLdesc
computerA,hard disk 95% full,4,Computer,ComputerSupport,Support,Sydney,https://www.ibm.com,Support link
computerB,hard disk 1 failure,6,Computer,ComputerSupport,Support,Melbourne,https://www.ibm.com,Support link
computerC,anti virus outdated,3,Computer,ComputerSupport,Support,Brisbane,https://www.ibm.com,Support link
computerD,Python not found,5,Computer,ComputerSupport,Support,Adelaide,https://www.ibm.com,Support link
Border Router,Temperature > 80,6,Network,NetworkSupport,Network,Sydney,https://www.ibm.com,NetworkSupport
Core Router,Interface 1 Port B down,5,Network,NetworkSupport,Network,Melbourne,https://www.ibm.com,NetworkSupport
Access Router,Interface 10 Port 200 utilisation > 90%,4,Network,NetworkSupport,Network,Brisbane,https://www.ibm.com,NetworkSupport
LDAP,Authentication failure ,4,Application,AppSupport,Application,Sydney,https://www.ibm.com,AppSupport
Security App,"Login failure more than 5 times, user: julius",5,Application,AppSupport,Application,Sydney,https://www.ibm.com,AppSupport
Payroll All,Database storage almost full (96.6%),6,Application,AppSupport,Application,Sydney,https://www.ibm.com,AppSupport

lightbulb
Depending on the Operating System where you created the spreadsheet and the Python script, you might want to convert the end-of-line character of the CSV file using the dos2unix command line.

We can use this Python script to read the CSV file and send it to the webhook integration.  Let us put the Python script into a file called sendCSV.py.

the content of sendCSV.py

# This script reads from a CSV, converts each row to JSON, and sends it to the CP4AIOps Webhook probe.

import sys
import json
import csv
import requests
import time

# The CP4AIops webhook endpoint. Please modify this to your environment
url = 'https://whconn-7c8dc065-839b-4ae8-99cd-1f9cd7185ee8-cp4aiops.apps.o2-182300.cp.fyre.ibm.com/webhook-connector/v91vf0g43gn'
headers = {
        'Content-Type': 'application/json',
        'Authorization': 'Basic anVsaXVzOldoYXRJc0FHb29kUGFzc3dvcmRGb3IyMDI1Pw=='
}

# Wait time between each send. It can be used to simulate delay for alerts received.
sleep_duration = 1.0  

if len(sys.argv)!=3 :
    print("Usage: python thisprogram.py <inputfile.csv> <outputfile.json>")
    sys.exit()
inf = sys.argv[1]
ouf = sys.argv[2]
print("Input CSV file: ",inf)
print("Output JSON file: ",ouf)

csv_file = csv.DictReader(open(inf, 'r'))

json_list = []
for row in csv_file:
    json_list.append(row)

open(ouf, 'w').write(json.dumps(json_list))

for row in json_list:
    print (row)
    response = requests.post(url , verify=False, headers=headers, json=row)
    print (response)
    time.sleep(sleep_duration)

After running the Python script...

% python sendCSV.py AlertsSpreadsheet.csv out.json
The out.json contains the payload in JSON format for auditing/troubleshooting purposes.

We get the following alerts in the CP4AIOps UI:

HolyGuacamole

Three alerts group and 10 alerts were created... Our policy is working.  Holy Guacamole! That was easy.

Summary

We have gone through the journey of creating alerts through Webhook integration, from using the out-of-the-box sevOne mappings to customising the payload to our needs and using a Python script to push the alerts created using a spreadsheet. At the same time, we have created a simple scope-based grouping policy as the target for alerts.

We can extend this fundamental principle and modify the script to drive other, more complex policies.  You can use your preferred programming language to create your test environment.  During a discussion with my customer who asked the initial question, we realised that the webhook mapping most likely stays the same, as in the production environment,  the alerts fields do not change that quickly. 

Hopefully, you will have more pleasant Holy Guacamole moments with CP4AIOps alerts.

0 comments
9 views

Permalink