Recently, a customer asked me, "In CP4AIOps, How do I generate alerts to test the automation policy I am creating?". While responding to the question, I thought the answer might also be helpful to others, hence this blog.

This tutorial will go through the process step-by-step using the following plan:

• First, we will deploy and configure the CP4AIOps (Cloud Pak for AI Operation) generic webhook. Many blogs exist, but not many go through the actual development of alert mapping.
• Second, we will use one out-of-the-box sample configuration and send a test event using CURL.  Some troubleshooting steps are included in case you have issues following this tutorial.
• Since the original purpose of generating the alerts is to test a policy, let's create a simple policy. We will configure a policy to produce a scope-based group.
• Next, we will modify the webhook JSONata to customise the payload; you most likely want a specific field, including some custom fields, as input for your policy.
• Generating JSON manually can be time-consuming. We need a way to create lots of alerts to test the policy quickly. The question at the beginning of this blog was asked to help the customers' business analysts test some policies. Most business analysts use spreadsheets, so we will create our alerts in a spreadsheet and save them as a CSV file.
• Finally, we will use a simple Python script to ingest the CSV file and test our automation policy.
  

The screenshots in this tutorial were taken on CP4AIOps version 4.8.x, the most recent version at the time of writing.

If you want to follow along on your environment, you will need the following:

• Access to CP4AIOps version 4.x.x (yes, it does not have to be the latest version) with a user role to edit policy and integration.
• Access to a workstation with CURL and Python that can access the CP4AIOps UI.

	If you have access to a traditional Netcool Omnibus system, another way is to inject the test alerts into Omnibus and let it replicate to CP4AIOps using the Netcool connector.   I have written a blog to use the Omnibus REST interface here.

Deploy a webhook integration.

Last year, I delivered training on CP4AIops. The students sent events to a generic webhook as part of the lab exercise. Upon seeing the alerts in the CP4AIOps console, one student had a lightbulb moment and shouted, “Aha! That was easy.” Remembering the contagious positive response, let us use a webhook probe to send our test alerts. 

But first, we need to configure our webhook integration. Yes, CP4AIOps supports multiple webhook probes running simultaneously, so we can create our webhook.

Follow the following steps:

1. Login to your AIOps system.
2. Select Integration from the menu.
3. Click the Add Integration blue button. If the Generic Webhook has been configured for your environment, you should click the Generic Webhook instead of steps 3 to 6 and then click Add Integration.
4. Type in “webhook” in the search bar.
   
5. Select the Generic Webhook card.
6. Click the Get Started option button.
7. Name the webhook integration instances. Let us use our name; make this unique and personal, as only we will use this integration. 
   
8. Specify a username and password. Paste this into a temporary text editor. We will need this to call the webhook.
9. Leave the generated webhook route as is.
   
10. Click Next. The fun part comes: We will map the incoming fields to the target alert fields.
11. Some predefined mappings are available; click the Load sample mapping to select one.
12. Let us select one with a single-layer JSON input. Simple is good. So choose the sevOne sample.
    
13. Study the generated JSONata configuration. We will not modify this JSONata yet; we want to test it first to ensure that our Generic probe runs OK—one step at a time.
14. Click save. Wait about 2 minutes until the Generic Webhook integration status changes to a green check mark and is Running.
15. When we click save, CP4AIOps spin a Generic Webhook pod; we will query this pod later during troubleshooting.
16. Copy the webhook route of the integration we just created using the blue copy-to-clipboard icon. This route is the URL to send your alerts' JSON payload. We will need it later, so paste it into our temporary text editor.
    
    
17. The webhook route should look like this:  https://whconn-7c8dc065-839b-4ae8-99cd-1f9cd7185ee8-cp4aiops.apps.o2-182300.cp.fyre.ibm.com/webhook-connector/v91vf0g43gn.
18. The webhook integration is now ready. 
19. In the next section, we will send a JSON payload. Before that, let's understand a bit more about the sample JSONata mapping.

	There are many possible sources of events feeding into CP4AIOps as alerts.  To name a few: From external Omnibus probes. From external Omnibus through Netcool connector. Generated as synthetic events through Impact connector. Through CP4AIOps alerts API. Through Kafka (depending on CP4AIOps version). Generated from CP4AIOps topology as resources status. From external sources through CP4AIOps webhook integration.
	This tutorial uses the last method, which generates the contagious positive response :D.

Test alerts

We will be sending a test alert using CURL.  To do that, we will need the following information:

1. The Webhook Endpoint URL.  We already have this from the Webhook route earlier.
2. Username and password. We will use basic authentication by decoding them in the CURL header.
3. The Actual JSON payload. We will store the payload in files because you can edit and reuse files. This is a better alternative to specifying the payload on the command line.

Authorization

We will use basic authentication by encoding our username and password using base64 by performing the following (I am using a standard RHEL 9.x workstation).

Note the use of -n in the first echo command; this removes any trailing newline character. The wrap 0 in the base64 command serves the same purpose. The last echo command makes the output more readable and makes it easier to copy the generated encoded value.

The resulting decoded string may not produce the desired result without these options.

JSON payload

We have purposely selected the sevOne sample webhook mapping, as it uses a flat JSON structure, which is easier to work with.  Let us create a file named sevOneEvent.json containing one sevOne alert:

Sending the event.

We now have everything to send the test event. Manually entering and correcting any long command can be challenging.  Putting the CURL command in a file and running it is more manageable for us as human beings (unless you are an Android).

Note the Status 200 status response, which tells us the command succeeded. If you do not see this, read the troubleshooting guide below.

If we check our alerts in CP4AIOps, we should see the event we just sent.

Woohoo!  Our webhook integration works.  It's time for some small dance—tra la la la la.

Troubleshooting

If your command does not work, then there are a few steps that you can do.

1. Run the curl command verbosely by adding the -v option to the curl command.
2. Check the log of the webhook integration pod.

The following are examples of errors as you run the CURL command.

Here are sample steps to check the log of the webhook integration pod.

• First, get the pod name.  We are searching for a pod with "webhook" in its name.  I am selecting the youngest pod because I have two webhook integrations, and I just created the webhook pod for this tutorial.

• Now, get the pod's logs. You might want to use the tail options to reduce the output to the last few lines of the log.

The above output shows a successful one. A log output with errors might look like the following (depending, of course, on the error):

The policy

We have tested the webhook integration, and it works. We can now proceed with customizing the JSON payload. 

Since the main problem we tried to address in this tutorial is to inject alerts to test a policy, we need to know the alert fields required by the policy. 

In this tutorial, we want to test a simple policy; a scope-based grouping policy is simple enough. A Scope-based policy will need at least one field as the scope. So, let us create the policy.  Following along the steps below:

• From the main menu, select Automations.
• Click the Create policy blue button.
• Click the Create word on the Group alerts based on scope.
• Enter the policy name and optional description in the Policy details section.
• You can enter any number for the execution order. I entered 88 to represent that our policy is relatively the least important; besides, in some cultures, 88 is a lucky number :D.
  
• The following section configures the policy triggers and specifies when the policy should be run. We select both the Before an alert is created and the After an alert has been updated when the lastOccurrenceTime changes. This means this policy is run when CP4AIOps detects new and existing alerts that had the lastOccurrenceTime updated since the last time the check was done. 

	Another alert field commonly used to verify the updated alerts is Severity. If we specify a change in the Severity policy trigger, we can manually run the policy by changing the alert's severity through the UI.

• Even though the policy is now running, we can specify one or more conditions that will act as a gatekeeper and perform a test on the alert's payload. We can specify zero or more conditions. It is recommended that we have at least one condition. For the condition here, we are only interested in processing the event from our webhook, so we are testing that the alert sender's name is Julius Webhook.   When we map our webhook integration later, we need to ensure that we assign this string.
  
• Now comes the policy's action. The policy's purpose is to group alerts together based on a scope.  We need to specify the scope. For this tutorial, let us choose the scope based on the alerts' custom field: details.scope. Select the details field, then type in the subfield custom as shown, and then click apply.  We are configuring CP4AIOps to search for alerts with the same alert.details.scope content, and group them together. Remember, like the sender name, we will need to include the details.scope in our We will keep our webhook integration.
  
• We will not modify the Time Window section, so go back to the top of the User Interface interaction, ensure the enable switch is set to enable and click Save.  Our policy is now ready.

Alerts mapping take 2

Earlier, we sent alerts manually through CURL with the sevOne payloads.  Now, we are creating our payload.  There are two compulsory fields expected by the policy we want to test: sender.name = "Julius Webhook", and there should be at least two alerts with the same details.scope.  We will put the sender.name directly as a literal string in our webhook mapping, so we only need to specify the same scope in our alerts payload. Let us create two files with the required JSON payload as follows:

With the new payload, we need to modify our webhook mapping accordingly.  Here is the new webhook mapping.

Now, let us send the content of the files through CURL as before.

And here is what the alerts screen should look like.

The alerts are grouped, and the correlation column shows they are scope-grouped. Yay, whoop whoop whoop.... 

	Note that we have hardcoded the expirySeconds of 300 in the webhook mapping. The purpose of our alerts is to test a policy, not to reside in the existing alerts forever. As a recommended practice, always think of the alert lifecycle clean-up phase. We do not want a bloated system, hence auto-cleaning your alerts.

More alerts, and then more...

We now need a way to let the business analysis specify more alerts. The easiest way to do this will be to use a spreadsheet and then save the content into a CSV.  Luckily, Python provides libraries that can read the content of a CSV file, convert it to JSON, and send it through HTTPS.

So, let us use a spreadsheet, and very quickly, we can create a sample of 10 alerts with the three overall scopes. In the real environment, we may want to study the actual alert of interest. We may want to replay those alerts by copying them into the spreadsheets. After all, we create and test our policy to be applied to real alerts.

Here is the spreadsheet's CSV content so that you can test it in your environment.

	Depending on the Operating System where you created the spreadsheet and the Python script, you might want to convert the end-of-line character of the CSV file using the dos2unix command line.

We can use this Python script to read the CSV file and send it to the webhook integration.  Let us put the Python script into a file called sendCSV.py.

After running the Python script...

We get the following alerts in the CP4AIOps UI:

Three alerts group and 10 alerts were created... Our policy is working.  Holy Guacamole! That was easy.

Summary

We have gone through the journey of creating alerts through Webhook integration, from using the out-of-the-box sevOne mappings to customising the payload to our needs and using a Python script to push the alerts created using a spreadsheet. At the same time, we have created a simple scope-based grouping policy as the target for alerts.

We can extend this fundamental principle and modify the script to drive other, more complex policies.  You can use your preferred programming language to create your test environment.  During a discussion with my customer who asked the initial question, we realised that the webhook mapping most likely stays the same, as in the production environment,  the alerts fields do not change that quickly. 

Hopefully, you will have more pleasant Holy Guacamole moments with CP4AIOps alerts.