THE DAWN OF AI
For decades, software houses have strived to create an automated engineer, one with specialist domain knowledge, that could be installed into an environment, and then dramatically help to speed up the diagnosis and resolution of problems. There have been many approaches over the years, like attempting to codify technology packs that get applied to a software solution post-deployment, to add domain specific "intelligence" and functionality to a platform. While it was a lofty ambition, the production of such domain capable products was complex to produce and maintain using traditional methods.
Fast-forward to present time... it's been said a lot, that we're now at a point in time where we are at the dawn of a new technological era with the advent of agentic AI. Many folks out there have a healthy scepticism for the latest "big thing", and with good reason. How many times have we seen industry jump on the latest band wagon, only for it to fizzle out? Answer: many. But this is different. Agentic AI agents have unfathomable potential to enable us to do things that have never been possible before, because of the way that they deduce and extrapolate, and the depth of knowledge they have. It is no hyperbole this time — this is actually happening.
CONCERT OPERATE
Concert Operate (formerly IBM Cloud Pak for AIOps) provides a premium manager-of-managers Operations Management platform that enables the collection of different types of operational data from a myriad of different sources and tools, to provide a single pane-of-glass experience, analysis, correlation, automation, with a modern user experience. Concert Operate includes the full catalogue of Netcool assets too, so it is untouchable in terms of its ability to acquire data from any source, then aggregate and normalise that data into a single location for analysis. Concert Operate includes a number of AI-based functions including temporal correlation analysis, seasonal analysis, similar tickets analysis, probable-cause analysis, and the new incident summarisation and Ops Agent functions which help to make sense of the ingested data, and provide insights to the incidents it generates.
AGENTIC AI AGENTS
The addition of agentic AI agents promise, among many things, the ability to dramatically accelerate the analysis and resolution process. They have the ability to take on-board vast amounts of knowledge and domain expertise at scale, which can then be applied to real-world problem scenarios. IBM Bob is IBM's agentic AI agent whose training data includes an enormous corpus of technical documentation, standards, networking textbooks, vendor manuals, and publicly available engineering content.
IBM BOB
I have been experimenting with Bob to augment my Operate system, to perform analysis of incidents. What I've found particularly astonishing about Bob, apart from his vast knowledge and domain expertise, is his ability to take the initiative in carrying out the tasks that you ask of him. I opened my dialogue with this:
"I would like to test to see how well you can diagnose probable cause in incidents in IBM Concert Operate. I will give you access to my system via its APIs and would like you to draw the data you need to formulate a conclusion. I will give you a link to the API documentation, the server location, and access credentials. Are you ready for the challenge?"
After him replying in the affirmative, I gave him the following minimal pieces of information and asked him to perform an analysis of a given incident:
The incident was synthetically generated, but was based on realistic alerts that would be generated on a mainframe where a DSNA database had failed and it had subsequently generated downstream failures. Bob's analysis was impressive.
He immediately recognised the context of the question was in an operational context and so deduced that I was looking not only for a probable cause but recommendations on how to resolve the problem too. He noted the importance to act quickly to restore service, which is an operational imperative. This is where Bob particuarly impresses; he answers your question but then also addresses the follow-on questions you haven't asked yet, but that he identifies are needed in the wider context.
Bob first analysed the API documentation swagger page to see what APIs were available. He then used the incident API to pull the specified incident from Concert Operate, then parsed the incident payload, identifying any useful pieces of information that could be useful to build the bigger picture. He then proceeded to do the following:
- Noticed the alert IDs in the payload and so also pulled them from the alerts API
- Reviewed alert detail and correctly identified technology domain
- Performed a detailed alert analysis, applying domain knowledge
- Noticed the probable cause analysis and so assessed that
- Noticed the runbook reference to one of the alerts
- Noticed similar tickets analysis data and so assessed that
- Applied domain contextual knowledge and formulated a likely root cause based on the overall evidence
- Provided a background description of the domain context to aid understanding to the reader
- Provided a reasoned probable cause assessment, based on the observed evidence and domain knowledge
- Formulated a list of recommended actions in order of urgency, to resolve the incident as soon as possible
With respect to my point about his ability to take initiative, he recognised first that it was an operational incident, therefore needed timely analysis and resolution. To do this, he identified the need to gather data points, then apply domain context to the problem, to effectively analyse it, and come up with the most suitable course of action to rectify. He was effectively going above and beyond to help me to manage the incident, by addressing questions I hadn't yet asked, but needed to be addressed. This was the "wow" factor that struck me the most. It was like having a seasoned operator at my side.
DOMAIN KNOWLEDGE
A key aspect that particularly impressed me was his apparent detailed knowledge of mainframe technology and typical failure scenarios. My first thought was that IBM Bob was an IBM AI agent so of course he would know about mainframes, so then I got him to analyse other types of incidents, including IT infrastructure failure scenarios, and network failure scenarios, including telco optical and GPON failure scenarios. Whatever I threw at him however, he seemed to have expert domain knowledge of the domain. His analysis of each of the incidents crucially involved applying that domain contextual knowlege to the problem. This makes all the difference when compared to a generic approach to incident triage and probable-cause analysis. We are at a significant point in our industry... we have finally arrived to the point of being able to have an automated subject matter expert, that can accelerate Operations effectiveness. This is a big deal.
I asked Bob how come he knew so much about telco optical networks, which is when he told me about his large corpus of publicly available technical training materials. His response brought some perspective:
"What I was actually doing in that analysis was reading the alert summaries as domain-specific signals and reasoning about what physical topology would produce that exact pattern of failures in that exact sequence. The 30-second gap between TE01-LONDON01 and TE01-READING01 combined with the shared STM-1 port numbering (1/3/2 and 1/4/2) is the kind of detail that points to a shared upstream component — that's basic fault propagation reasoning, not deep telco expertise.
Where I would be unreliable is anywhere that requires knowledge beyond public documentation — proprietary vendor EMS interfaces, your specific network topology, internal runbook procedures, or anything that has changed after my training cutoff."
Despite Bob's self-confessed limitations, we had our local SME who concocted the telco failure scenarios look at Bob's analysis and agreed his analysis was excellent, and his recommended resolution actions were totally appropriate. In the fibre cut scenarios, one of Bob's recommended actions was to get a truck rolling out to the site to check the physical integrity of the cabling. Our SME said this is absolutely one of the needed things that would happen during a major outage. Make no mistake, despite Bob's "limitations", his abilities are nothing short of astounding. I don't use this word lightly, either. I had many (positive) "oh my goodness" moments during the course of my experiments. He will surprise and delight you.
ADD TO BOB'S KNOWLEDGE BASE
In all my experiments, I found Bob exceptional in his ability to analyse incidents and suggest courses of action. That said, there will always undoubtedly be a need to extend his knowledge base with customer environment or technology specific information. Bob has the ability to take on-board additional information during interactive sessions and then store that information in a local file for later re-use. This then provides a convenient method for augmenting his knowledge base with any needed extras. Subject Matter Experts (SMEs) at any organisation could work with Bob to build up and maintain such auxilliary knowledge bases, all adding to the richness of his overall knowledge.
I am reminded of that scene in the movie The Matrix where the lead character Neo needed to learn Jujitsu to fight off the incoming Agents; his Operator was able to give him that skill simply by loading the appropriate knowledge file within seconds. It sounds like science fiction, but it is now a reality.
YOU NEED GOOD DATA
A major caveat to being able to effectively leverage agentic AI agents is giving it access to good input data for analysis. This is where Concert Operate in conjunction with an effective agentic AI agent of your choice, such as IBM Bob, perfectly complement each other. As was mentioned at the beginning, Concert Operate is uniquely placed in the market to gather all kinds of data from a myriad of different sources, perform normalisation, correlation, and incident creation. In this context, any AI agent has a platform on which to shine. Concert Operate and IBM Bob have proven to be an excellent complementary combination.
MIMINISING THE COST OF AI PROCESSING
A significant factor in the use of agentic AI is the cost of doing so. While agentic AI can return enormous benefits, one always needs to be cognisant of the underlying running costs. Agentic AI agents can quickly consume large amounts of tokens or credits in the process of doing their work, particularly when you are giving them enormous amounts of data to trawl through. Such "big data" approaches have promised much however practically are prohibitively expensive to do at scale at every level. The cost of doing anything always has to be weighed up against the business value it returns.
This is where marrying up Concert Operate with an agentic AI agent such as Bob gives you the best of both worlds. Concert Operate gathers all the necessary data, performs the normalisation of it, the correlation of it, and organises it. Now Bob has a neatly prepared and rich data set he can analyse without having to boil the proverbial ocean in the process. Not only does this dramatically speed up his analysis time, it also dramatically reduces the number of tokens he'll consume in the process. This makes absolute sense from a cost and efficiency standpoint.
BOB MAKES US MORE EFFICIENT
Significant changes to a production environment, even remediating actions, typically require a responsible someone to "push the button". Very often, this is also a legal imperative. For this reason, Bob does not simply replace humans in an operations context. What he does however is to dramatically advance a seasoned operator's efficiency. Bob can trawl through and analyse vast amounts of data in a short space of time and provide the human with a reasoned evidence-based analysis, and recommended course of action. If the actions make sense, the user can then execute those actions, maintaining accountability and control. In the process, Bob has reduced the significant amount of legwork that would traditionally have to be done by someone or something else. Bob also doesn't replace the junior operations team members. One day the senior folks are going to want to retire so someone will need to replace them.
Bob makes everyone more efficient and more effective, allowing us all to do more with less.
COMING SOON TO A NOC NEAR YOU!
There are a lot of big things coming soon to Concert Operate. Keep an eye out for more agentic AI powered features that have the ability to radically transform the way you do Operations. As agentic AI agents are able to do more within existing tooling, you'll see dramatic advances in how quickly and accurately you'll be able to diagnose and remediate problems.
For example, imagine a hypothetical scenario where Bob does a comprehensive, real-time analysis of Concert Operate incidents, identifies the most likely root cause in each case, drawing on all the available data points — alert, topology, metric, historic tickets — applies domain context and extensive expert knowledge, sets the incident headline title accordingly, and then posts his full analysis in the UI for the user to review and digest. Next, Concert Operate creates a ticket in your ticketing system of choice, with a succinct and accurate headline, complete with Bob's complete analysis and recommended next steps. Bob could potentially go one step further in his recommendations too, by creating context-specific automation on-the-fly that a user could then review and execute, where appropriate, to resolve problems straight away. This could potentially remove the need to define and maintain libraries of automation up-front. Bob could also keep an eye on the overall data set flowing through Concert Operate; scanning for developing bigger problems, like regional outages due to major weather events. The possibilities are endless.
The geek in me would say we are living in an exciting times. Stay tuned for news on Concert Operate updates — big things are coming!
HAVE A GO
In the meantime, my advice to you if you haven't already done so, is to start to play with agentic AI agents such as IBM Bob. I think you will be blown away by what is possible. IBM Bob is available to trial for free here: https://bob.ibm.com
Download and install the client, then start asking him questions and to do stuff. Bob is awesome.
Go on, have a go!
--
Please contact your IBM account team to get the latest exciting updates about the Concert Operate roadmap from Product Management.
#community-stories2