AIOps

AIOps

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.


#ITAutomation
#AIOps
#CloudPakforAIOps
#AIOps

 View Only

Cloud Pak for AIOps 4 tips: housekeeping for incidents using Netcool/Impact

By Zane Bray posted 04/10/26 07:16 AM

  
image

IBM Cloud Pak for AIOps (AIOps) has an in-built housekeeping mechanism for alerts by providing an attribute expirySeconds that will automatically clear alerts whose last occurrence timestamp is older than expirySeconds ago. But what about housekeeping for incidents? What if we have a requirement whereby any created incident that has not been updated within 7 days should be automatically resolved?

This blog outlines a simple automation to enable this feature in AIOps and will automatically set the state of all incidents whose last update timestamp (lastChangedTime) is older than the specified number of days ago. Note that it requires that you have Netcool/Impact (Impact) deployed as it uses Impact to perform this task via a Policy Activator Service.

CREATE A ZENAPIKEY

The automation in Impact will make a connection to AIOps to carry out its housekeeping activities. To do this therefore, you need to create an API key in AIOps and use it in the housekeeping automation. You can follow the instructions provided in this previous blog to create your API key. You may already have one created previously, in which case you can just use the existing one.

CREATE AN IMPACT POLICY

Create the following Impact-Policy-Language (IPL) policy in your Impact system:

////////////////////////////////////////////////////////////////////////
//
// AIOps Incidents Housekeeping - AIOPS_HousekeepIncidents
//
// Created by: Z. Bray (zane) 30/03/2026
//
// The purpose of this policy is to check the Incidents list in AIOps
// and automatically resolve ones whose lastChangedTime is older than
// the specified amount of time. This is to ensure that there is not
// a build-up of old incidents in the system.
//
// The default value is 7 days. This should be modified below to suit.
//
// This policy should be run by a Policy Activator service once an hour.
// 
////////////////////////////////////////////////////////////////////////

// DEFINE VARIABLES - MODIFY TO SUIT REQUIREMENTS

RetentionDays = 7;
RetentionSeconds = RetentionDays * 86400;
TargetState = '{"state":"resolved"}';

////////////////////////////////////////////////////////////////////////

Log("Housekeep Incidents: Looking for incidents with lastChangedTime older than " + String(RetentionDays) + " day(s) ago ...");

// DEFINE AIOPS CONNECTION PARAMETERS
HTTPHost="cpd-aiops.apps.aiops-4.cp.fyre.ibm.com";
HTTPPort=443;
Protocol="https";
ChannelKey="";
FilesToSend=null; 
FormParameters=null;
Method="GET";
AuthHandlerActionTreeName=null; 

HeadersToSend = NewObject();
HeadersToSend['Content-Type']='application/json';
HeadersToSend['X-TenantID']='cfd95b7e-3bc7-4006-a4a8-a73a79c71255';
HeadersToSend['Authorization']='ZenApiKey ABCDEFGabcdefg12345678906cWhuVUh5aFFQd2xwY044bkhWdGRFZFhqCg==';

HttpProperties=NewObject();
HttpProperties.AlwaysSendAuth=true;
HttpProperties.TrustCertificate=true;

Path = "/aiops/api/issue-resolution/v1/incidents";

// GET INCIDENT LIST
Log("Housekeep Incidents: Retrieving incidents list...");

y = GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method, AuthHandlerActionTreeName, FormParameters, FilesToSend, HeadersToSend, HttpProperties);
Log("Housekeep Incidents: Response for incident list retrieval: " + ErrorReason + " " + ResultCode);

if (ResultCode == 200) {

    // CHANGE METHOD TO PATCH TO UPDATE ANY ALERTS
    Method="PATCH";

    // SET PAYLOAD FOR INCIDENT UPDATE
    HttpProperties.Content = TargetState;

    // PARSE RESULT SET
    ParsedIncidents = ParseJSON(y);

    Log("Housekeep Incidents: Got " + Length(ParsedIncidents.items) + " incidents");

    // INITIALISE LOOP VARIABLES
    i = 0;
    j = 0;

    // ITERATE OVER RESULT SET
    while (i < Length(ParsedIncidents.items)) {

        // CONVERT TIMESTAMP TO UTC SECONDS AND CALCULATE DELTA
        MyDate = RExtract(ParsedIncidents.items[i].lastChangedTime, "(....-..-..).*");
        MyTime = RExtract(ParsedIncidents.items[i].lastChangedTime, "....-..-..T(..:..:..).*");
        NewTimestamp = MyDate + " " + MyTime;
        ConvertedTimestamp = ParseDate(NewTimestamp, "yyyy-MM-dd HH:mm:ss");
        Delta = GetDate() - ConvertedTimestamp;

        Log("Housekeep Incidents: Checking: " + String(i) + ": " + ParsedIncidents.items[i].id + " with lastChangedTime: " + ParsedIncidents.items[i].lastChangedTime);

        // CALCULATE IF MORE THAN THE RETENTION PERIOD
        // MARK AS RESOLVED IF DELTA IS MORE THAN THE RETENTION PERIOD
        if (Delta > RetentionSeconds) {

            Log("Housekeep Incidents: Incident has NOT been updated more recently than " + String(RetentionDays) + " day(s) ago");
            Log("Housekeep Incidents: Resolving incident with id: " + ParsedIncidents.items[i].id + ": " + ParsedIncidents.items[i].lastChangedTime + " ...");
          
            Path = "/aiops/api/issue-resolution/v1alpha1/incidents/" + ParsedIncidents.items[i].id;

            z=GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method, AuthHandlerActionTreeName, FormParameters, FilesToSend, HeadersToSend, HttpProperties);
            Log("Housekeep Incidents: Response for: " + ParsedIncidents.items[i].id + ": " + ErrorReason + " " + ResultCode);
            
            j = j + 1;
        }

        i = i + 1;
    }

} else {
    
    Log("Housekeep Incidents: unable to GET incident list.");
}

Log("Housekeep Incidents: Resolved " + String(j) + " incidents");

NOTES:

  • Modify the header comments as needed.
  • Modify the RetentionDays variable to the number of days you want to keep incidents until they're automatically expired. The default is 7 days.
  • Update HTTPHost with the FQDN of your AIOps instance.
  • Update the Authorization property with your ZenApiKey. Note the word: "ZenApiKey" is needed as part of the authorisation property.
  • Add any further log statements as desired.
  • Save the policy with a suitable name - eg. AIOPS_HousekeepIncidents
  • Note that the policy uses the Impact function ParseDate to calculate the last time the incident was updated. It does not take into account time zone however this is likely to be unimportant. If it is important, the Delta value it calculates can be adjusted with an appropriate number of seconds.
  • This policy sets old incidents to a state of Resolved, which will also have the effect of clearing any underlying alerts. The default housekeeping automation in AIOps will then move the incident to closed some minutes later. If any of the underlying alerts recur in the meantime however, be aware that the incident will reopen. This is expected behaviour.

CREATE A NEW POLICY ACTIVATOR SERVICE

Create a new Policy Activator Service in Impact to run your new policy. A suggested run interval is once a day (86400 seconds).

image

NOTE: It is recommended to check the box: "Starts automatically when server starts"

Save your new service and then click the green "Play" button to start it running.

--

Congratulations, you now have an automation that will automatically resolve any incidents that haven't been updated more than your specified number of days ago.

See the IT & Network Automation Tiger Team website for more tips-and-tricks blogs on AIOps and Netcool.

0 comments
23 views

Permalink