IBM Cloud Pak for AIOps (AIOps) has an in-built housekeeping mechanism for alerts by providing an attribute expirySeconds that will automatically clear alerts whose last occurrence timestamp is older than expirySeconds ago. But what about housekeeping for incidents? What if we have a requirement whereby any created incident that has not been updated within 7 days should be automatically resolved?
This blog outlines a simple automation to enable this feature in AIOps and will automatically set the state of all incidents whose last update timestamp (lastChangedTime) is older than the specified number of days ago. Note that it requires that you have Netcool/Impact (Impact) deployed as it uses Impact to perform this task via a Policy Activator Service.
CREATE A ZENAPIKEY
The automation in Impact will make a connection to AIOps to carry out its housekeeping activities. To do this therefore, you need to create an API key in AIOps and use it in the housekeeping automation. You can follow the instructions provided in this previous blog to create your API key. You may already have one created previously, in which case you can just use the existing one.
CREATE AN IMPACT POLICY
Create the following Impact-Policy-Language (IPL) policy in your Impact system:
////////////////////////////////////////////////////////////////////////
//
// AIOps Incidents Housekeeping - AIOPS_HousekeepIncidents
//
// Created by: Z. Bray (zane) 30/03/2026
//
// The purpose of this policy is to check the Incidents list in AIOps
// and automatically resolve ones whose lastChangedTime is older than
// the specified amount of time. This is to ensure that there is not
// a build-up of old incidents in the system.
//
// The default value is 7 days. This should be modified below to suit.
//
// This policy should be run by a Policy Activator service once an hour.
//
////////////////////////////////////////////////////////////////////////
// DEFINE VARIABLES - MODIFY TO SUIT REQUIREMENTS
RetentionDays = 7;
RetentionSeconds = RetentionDays * 86400;
TargetState = '{"state":"resolved"}';
////////////////////////////////////////////////////////////////////////
Log("Housekeep Incidents: Looking for incidents with lastChangedTime older than " + String(RetentionDays) + " day(s) ago ...");
// DEFINE AIOPS CONNECTION PARAMETERS
HTTPHost="cpd-aiops.apps.aiops-4.cp.fyre.ibm.com";
HTTPPort=443;
Protocol="https";
ChannelKey="";
FilesToSend=null;
FormParameters=null;
Method="GET";
AuthHandlerActionTreeName=null;
HeadersToSend = NewObject();
HeadersToSend['Content-Type']='application/json';
HeadersToSend['X-TenantID']='cfd95b7e-3bc7-4006-a4a8-a73a79c71255';
HeadersToSend['Authorization']='ZenApiKey ABCDEFGabcdefg12345678906cWhuVUh5aFFQd2xwY044bkhWdGRFZFhqCg==';
HttpProperties=NewObject();
HttpProperties.AlwaysSendAuth=true;
HttpProperties.TrustCertificate=true;
Path = "/aiops/api/issue-resolution/v1/incidents";
// GET INCIDENT LIST
Log("Housekeep Incidents: Retrieving incidents list...");
y = GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method, AuthHandlerActionTreeName, FormParameters, FilesToSend, HeadersToSend, HttpProperties);
Log("Housekeep Incidents: Response for incident list retrieval: " + ErrorReason + " " + ResultCode);
if (ResultCode == 200) {
// CHANGE METHOD TO PATCH TO UPDATE ANY ALERTS
Method="PATCH";
// SET PAYLOAD FOR INCIDENT UPDATE
HttpProperties.Content = TargetState;
// PARSE RESULT SET
ParsedIncidents = ParseJSON(y);
Log("Housekeep Incidents: Got " + Length(ParsedIncidents.items) + " incidents");
// INITIALISE LOOP VARIABLES
i = 0;
j = 0;
// ITERATE OVER RESULT SET
while (i < Length(ParsedIncidents.items)) {
// CONVERT TIMESTAMP TO UTC SECONDS AND CALCULATE DELTA
MyDate = RExtract(ParsedIncidents.items[i].lastChangedTime, "(....-..-..).*");
MyTime = RExtract(ParsedIncidents.items[i].lastChangedTime, "....-..-..T(..:..:..).*");
NewTimestamp = MyDate + " " + MyTime;
ConvertedTimestamp = ParseDate(NewTimestamp, "yyyy-MM-dd HH:mm:ss");
Delta = GetDate() - ConvertedTimestamp;
Log("Housekeep Incidents: Checking: " + String(i) + ": " + ParsedIncidents.items[i].id + " with lastChangedTime: " + ParsedIncidents.items[i].lastChangedTime);
// CALCULATE IF MORE THAN THE RETENTION PERIOD
// MARK AS RESOLVED IF DELTA IS MORE THAN THE RETENTION PERIOD
if (Delta > RetentionSeconds) {
Log("Housekeep Incidents: Incident has NOT been updated more recently than " + String(RetentionDays) + " day(s) ago");
Log("Housekeep Incidents: Resolving incident with id: " + ParsedIncidents.items[i].id + ": " + ParsedIncidents.items[i].lastChangedTime + " ...");
Path = "/aiops/api/issue-resolution/v1alpha1/incidents/" + ParsedIncidents.items[i].id;
z=GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method, AuthHandlerActionTreeName, FormParameters, FilesToSend, HeadersToSend, HttpProperties);
Log("Housekeep Incidents: Response for: " + ParsedIncidents.items[i].id + ": " + ErrorReason + " " + ResultCode);
j = j + 1;
}
i = i + 1;
}
} else {
Log("Housekeep Incidents: unable to GET incident list.");
}
Log("Housekeep Incidents: Resolved " + String(j) + " incidents");
NOTES:
- Modify the header comments as needed.
- Modify the
RetentionDays variable to the number of days you want to keep incidents until they're automatically expired. The default is 7 days.
- Update
HTTPHost with the FQDN of your AIOps instance.
- Update the
Authorization property with your ZenApiKey. Note the word: "ZenApiKey" is needed as part of the authorisation property.
- Add any further log statements as desired.
- Save the policy with a suitable name - eg.
AIOPS_HousekeepIncidents
- Note that the policy uses the Impact function
ParseDate to calculate the last time the incident was updated. It does not take into account time zone however this is likely to be unimportant. If it is important, the Delta value it calculates can be adjusted with an appropriate number of seconds.
- This policy sets old incidents to a state of Resolved, which will also have the effect of clearing any underlying alerts. The default housekeeping automation in AIOps will then move the incident to closed some minutes later. If any of the underlying alerts recur in the meantime however, be aware that the incident will reopen. This is expected behaviour.
CREATE A NEW POLICY ACTIVATOR SERVICE
Create a new Policy Activator Service in Impact to run your new policy. A suggested run interval is once a day (86400 seconds).
NOTE: It is recommended to check the box: "Starts automatically when server starts"
Save your new service and then click the green "Play" button to start it running.
--
Congratulations, you now have an automation that will automatically resolve any incidents that haven't been updated more than your specified number of days ago.
See the IT & Network Automation Tiger Team website for more tips-and-tricks blogs on AIOps and Netcool.