Automating OpenShift Troubleshooting with IBM Bob Custom Modes

From repetitive log analysis to AI-guided debugging using a single YAML file.

Every OpenShift Engineer Has Been Here

• Whether you're a developer or an SRE, debugging applications on OpenShift often begins the same way. A pod enters CrashLoopBackOff, an alert is triggered, or users report failures in production. Before anyone can think about fixing the issue, the first task is always the same collect enough evidence to understand what is happening.
  That usually means switching between terminals, dashboards and documentation while running a familiar sequence of oc commands. Engineers inspect pod events, deployment configurations, restart history, previous container logs and resource limits before they can even begin forming a hypothesis. Although every production incident is different, the investigation process itself is surprisingly repetitive.
• As I explored IBM Bob Custom Modes, I started wondering whether this investigative process not just the commands could be encoded into a reusable workflow. Instead of asking an AI to execute individual commands, could it think through an investigation the same way an experienced OpenShift engineer would?

IBM Bob Custom Modes: More Than a Better Prompt

• One of the most interesting capabilities in IBM Bob is Custom Modes. At first glance they appear to be reusable prompts, but they are much more powerful than that. A Custom Mode defines the role Bob should adopt, when it should be used, which tools it can access and, most importantly, the workflow it should consistently follow.
  
  
• Rather than repeating the same troubleshooting instructions in every conversation, the knowledge is stored once inside a custom_modes.yaml file. Every time the mode is selected, Bob follows the same reasoning process and operational workflow.
  
  

Introducing the OpenShift Ops Assistant

• To explore what Custom Modes could achieve in a real operational scenario, I built the OpenShift Ops Assistant.
  The objective wasn't simply to generate oc commands from natural language. Existing AI assistants are already capable of doing that. Instead, I wanted to capture the troubleshooting process itself ,the sequence of checks, the evidence collection strategy, the error analysis and the reporting format that experienced SREs naturally follow.
  Developers can simply ask questions such as 'Why is my payment-service pod crashing?' or 'Find transaction ID abc-123 in the orders namespace.' Behind the scenes, Bob executes the appropriate investigation workflow, analyzes the collected information, detects common OpenShift error patterns and produces a structured report instead of raw command output.
  The result is an assistant that reduces the repetitive work involved in troubleshooting while allowing engineers to stay entirely within VS Code.

• One of the biggest challenges in OpenShift troubleshooting is that the information required to diagnose an issue rarely exists in one place. Engineers often need to inspect current logs, previous container logs, pod events, deployment configuration, restart history and resource limits before the root cause becomes clear. The OpenShift Ops Assistant automates this evidence gathering and correlates the results into a single analysis.
  
  
  • 🔍 Multi-Pod Log Search - Find transaction IDs across all pods instantly, trace requests through microservices
  • 🚨 CrashLoop Diagnosis - Automated root cause analysis for failing pods with specific fix recommendations
  • 🧠 Smart Error Detection - 6 categories of error patterns (memory, connection, auth, app, config, database)
  • 📊 Structured Reports - Professional analysis with severity levels, log evidence, and actionable next steps
  • 🎯 Interactive Workflows - Guided pod/namespace selection, no need to remember complex commands
  • ⚡ Fast & Efficient - Direct oc CLI integration, analyzes thousands of log lines in seconds
  • 🔒 Safe Operations - Confirmation required before any destructive actions
    
    
• The mode also lowers the learning curve for engineers who are less familiar with OpenShift. Instead of remembering complex oc commands or the correct investigation sequence, users simply describe the problem while the assistant determines the appropriate workflow.One interesting realization while building this mode was that troubleshooting is rarely about finding a single error message. Most production incidents require correlating information from multiple sources before a clear picture emerges. That insight heavily influenced the design of this mode it prioritizes gathering evidence first and drawing conclusions only after enough context has been collected.
• The result is an assistant that performs the repetitive investigation on behalf of the engineer, allowing developers to spend less time collecting evidence and more time understanding and solving the actual problem.
• Finally, every debugging session ends with a structured report containing severity, supporting log evidence, recommendations and suggested next steps. Instead of scrolling through thousands of log lines, engineers receive concise and actionable findings.
  
  

OpenShift Troubleshooting in Practice

• • Imagine a developer notices that a payment-service pod repeatedly enters CrashLoopBackOff and simply asks Bob why it keeps crashing.
    Instead of returning a handful of suggested commands, the OpenShift Ops Assistant validates cluster connectivity, gathers pod information, retrieves current and previous logs, inspects pod events, checks deployment configuration and reviews resource limits before attempting a diagnosis.
    
    
•  
• When asked to generate a health report of a container (this is a snapshot of detailed report generated by BOB)

The following health report illustrates how the assistant consolidates cluster state, resource utilization, service dependencies, and runtime metrics into a single operational summary rather than requiring engineers to inspect each resource individually.

• Another snapshot of when user asks to generate a log summary of a particular container

The log analysis report demonstrates another important capability. Instead of returning raw log lines, the assistant identifies recurring patterns, summarizes the findings, assigns severity levels, and highlights actionable recommendations.
 

Note: These are just this is a snapshot of detailed report (markdown file) generated by BOB.

Likewise, it can identify authentication failures, configuration problems, connectivity issues or common database errors using the built-in error pattern library.The goal is not simply to automate commands , it is to automate the investigation itself.

How It's Built

One design goal was to keep the implementation as lightweight as possible. Rather than introducing an MCP server, custom backend or additional services, the entire solution is implemented as a single custom_modes.yaml file.
The mode defines Bob's role as an experienced OpenShift SRE, specifies when the mode should be activated and embeds troubleshooting workflows, error pattern detection and reporting instructions directly into the configuration. Using Bob's built-in command execution capability, it interacts with the standard oc CLI, meaning it works anywhere the OpenShift CLI works.

Developer
      │
Natural language prompt
      │
      ▼
 IBM Bob Custom Mode
      │
 execute_command
      │
      ▼
 OpenShift CLI
      │
      ▼
 OpenShift Cluster
      │
      ▼
 Logs • Events • Resources
      │
      ▼
 Structured Analysis

Keeping everything inside YAML also makes the solution easy to maintain. Teams can review changes through pull requests, version operational knowledge alongside application code and customize workflows without maintaining additional infrastructure.

Getting Started

Getting started requires IBM Bob in Advanced Mode, the OpenShift CLI and access to an OpenShift cluster. Clone the repository, copy the provided custom_modes.yaml file into your project's .bob directory, reload VS Code and select the OpenShift Ops Assistant from Bob's mode selector. From there, troubleshooting becomes as simple as describing the issue in natural language.

Although this implementation focuses on OpenShift, the same concept can be applied to many other operational domains. Deployment validation, API governance, infrastructure diagnostics, security reviews, compliance checks, and platform operations all follow repeatable engineering workflows that can be encoded as Custom Modes. Instead of repeatedly explaining those workflows to an AI or documenting them in lengthy runbooks-they become reusable operational playbooks that every engineer can leverage.

Refer: https://github.ibm.com/yash-raj/Openshift-ops-assistant