Unify Operations with Shape Correlation in AIOps
Earlier this year, you may have seen this article by Chris Bailey: How IBM Cloud Pak for AIOps delivers unified operations management to the connected enterprise. The article illustrates the challenges that enterprises have with multiple expert-specific tools, and how Unified Operations Management can unify visibility, fault, and incident management as well as teams across the connected enterprise.
The first step to unify data is to have multiple integrations and the means of ingesting the data from those multiple systems. AIOps has many out of the box integrations and APIs to get data in.
The next step is to group or correlate the data across sources and domains so that it can be presented as one source of truth. Correlating data has multiple benefits:
- Efficiency Gains: By grouping related Alerts, operations teams face fewer tickets and less noise.
- Improved Context: Seeing correlated issues together helps teams diagnose problems faster.
- Impact Awareness: Understanding that an Alert is part of a broader issue aids prioritization.
- Reduced Duplication: One team member can address a root cause instead of multiple people chasing symptoms.
In AIOps, we have multiple means of grouping data. Whether through Topological Merge Rules, or using one of the Alert Grouping algorithms, we have now added another means to correlate data – Shape Correlation.
Why Shape Correlation?
It started with a simple problem. A host had a weird and obvious series of CPU Wait-time spikes and IO spikes. We knew it must be coming from something that was running on that host. The problem was there were many things running on it, and all we wanted to do was to find the one container which showed a similar metric pattern as those spikes. After manually inspecting all the timeseries, which was time consuming, the cause was found. Now with Shape Correlation this is done automatically!
Shape Correlation analyses metric (timeseries) data and correlates timeseries that have a similar shape. It can discover correlations from different resources and across domains and data sources. The creation of this algorithm was the result of close collaboration with IBM Research, and a thorough analysis of the state of the art, with a clear focus on being efficient and low-cost. The result, is that Shape Correlation requires no additional hardware as it is fast and efficient regardless of how many millions of timeseries you are analysing.
Real-World Examples
Consider metrics like Garbage Collection Rate and Garbage Collection Times Run. On a normalized chart these look to be one timeseries, but the tooltip shows they do indeed have different values. Shape Correlation automatically identifies and groups these timeseries—no manual configuration or tuning is required.
In the below example, multiple timeseries with similar shape are correlated – ensuring we have one problem for one person, rather than multiple tickets for multiple people to look at.
Final Thoughts
Shape Correlation gives Operations one more way to unify their data from disparate tools. For teams looking to reduce noise, improve MTTR, and gain deeper insight into system behavior, this feature is a must-try!