Introduction

Kubernetes incident investigation is rarely a single-command activity.

When a service shows readiness failures, restarts, latency, Kafka lag, or memory pressure, an engineer usually has to move across many sources: pod status, pod descriptions, namespace events, logs, metrics, runbooks, and historical context. The hard part is not only collecting this data. The harder part is keeping the investigation bounded, safe, evidence-grounded, and reproducible.

I wanted to explore whether Claude Code could be used not just as a coding assistant, but as an operational investigation harness: a structured way to guide an AI assistant through a real incident workflow without giving it broad, unsafe production access.

The result was Claude Ops Investigator, a context-aware Kubernetes incident investigation assistant built around Claude Code, MCP, read-only operational tools, evidence references, subagents, hooks, and structured reports.

Overall architecture

The design separates the agent harness from the operational application boundary.

Claude Code provides the workflow layer: slash command, coordinator, specialist subagents, hooks, skills, and rules.

The actual operational capability lives behind MCP: typed tools for Kubernetes, Prometheus, IBM Cloud Logs, runbooks, and evidence retrieval.

At a high level, Claude Code is not being asked to invent operational commands. It is being asked to reason through a controlled interface.

That distinction matters. Before we describe the ideas behind the tool development, let's take a look at an example investigation to whet your appetite.

Example investigation: readiness probe failures

Suppose we want to investigate readiness probe failures. Let's fire up Claude Code and enter the slash command.

/investigate-incident namespace=si service=multi-system-processor symptom="readiness probe failures during recent rollout" since_minutes=60

Claude Code gets to work:

and after a while finished investigation:

The investigation checked:

current pods
recent namespace events
pod descriptions
Prometheus restart metrics
IBM Cloud Logs probe failures
IBM Cloud Logs errors
runbooks

and produced the final report.

The problem: incident triage is noisy and risky

In a typical incident, an engineer may ask:

• Why did this service fail readiness?
• Did the pod restart?
• Was there an OOM?
• Did Prometheus show a spike?
• Are the logs showing probe failures?
• Is this related to a known runbook?

The manual workflow often means switching between:

kubectl
Prometheus
IBM Cloud Logs
runbooks
previous investigation notes
service catalogs

That creates several problems:

1. Repetitive work
2. Inconsistent investigation quality
3. Copy/paste-heavy evidence collection
4. Risk of unsafe shell commands
5. Reports that make claims without clear evidence
6. Context overload when raw logs are pasted into the assistant

The goal was to build a workflow where the assistant could help investigate, but only through safe, typed, read-only tools.

In the manual workflow, the engineer is responsible for collecting, filtering, and synthesizing everything. In the guided workflow, Claude Code follows a defined investigation path using typed MCP tools and returns an evidence-grounded report rather than an ad hoc note.

The core idea: separate the app from the harness

The most important design decision was to keep the actual investigation capability outside Claude Code-specific instructions.

The durable application layer consists of the MCP server, typed tools, evidence store, runbook catalog, service catalog, structured incident schemas, and automated validation tests. Claude Code acts as the harness around that application, handling the UX/workflow layer like slash commands, agent coordination, hooks, and final report formatting.

This separation avoids building an assistant that only works because of one brittle prompt file. The assistant works because the operational actions are exposed as typed tools with safe behavior and structured outputs.

Project structure

GitHub Repository: github.com/Randhir123/claude-ops-investigator

src/claude_ops/
  mcp/
    server.py              # Houses the FastMCP/ASGI server definitions
  tools/
    k8s_tools.py           # Wraps kubernetes-client; provides restricted list/describe tools
    prometheus_tools.py    # Standardizes specialized PromQL metric delta requests
    ibm_logs_tools.py      # Connects log engine queries for targeted regex match groups
    runbook_tools.py       # Interfaces with Vector DB / local Markdown runbook matches
  evidence/
    store.py               # Manages stateful trace logging and artifact file caching
  schemas/
    incident_report.py     # Strictly models target validation shapes for output verification

.claude/
  commands/                # Custom interactive slash commands (e.g., /investigate-incident)
  agents/                  # System definitions for the coordinator and specialist subagents
  hooks/                   # PreToolUse, PostToolUse, and Stop intercept lifecycle rules
  rules/                   # Execution constraints (e.g., handling metric gaps explicitly)

MCP tools: narrow, typed, and read-only

Instead of giving the assistant general shell access, I exposed specific tools through MCP:

k8s_list_pods
k8s_describe_pod
k8s_get_recent_namespace_events
k8s_get_pod_logs
k8s_top_pods

prom_get_pod_restart_increase
prom_get_pod_memory_usage
prom_get_pod_cpu_usage
prom_get_http_error_rate
prom_get_latency_p95
prom_ensure_connection

ibm_logs_search_probe_failures
ibm_logs_search_errors
ibm_logs_search_text

runbook_search
evidence_get_detail

This is much safer than asking an agent to generate arbitrary commands.

Instead of relying on a freeform shell execution like:

kubectl describe pod ...

the assistant explicitly calls:

k8s_describe_pod(namespace, pod_name)

Instead of building raw PromQL queries on the fly, it prefers pre-typed Prometheus tools such as:

prom_get_pod_restart_increase(namespace, service, since_minutes)

The assistant is not being asked to invent the operational interface; it is being asked to reason using a safe, sandboxed operational interface.

Evidence references instead of raw context flooding

A major issue with AI-assisted operations is context flooding. Raw pod descriptions, logs, events, and metrics can be large. If everything is pasted directly into the assistant context, the session becomes noisy, expensive, and quickly hits token limits. It also increases the risk of leaking sensitive details into intermediate logs or reports.

To solve this, the project uses an evidence model:

• Tool returns: A compact summary and a unique evidence_ref pointer.
• Raw output: Stored separately within the artifacts/ directory.
• Assistant: Reasons from the summary first, calling evidence_get_detail only when deep debugging is required.

A structured finding should look like this:

The target pod had no restarts during the incident window.
Evidence: ev_20260709T080831Z_8ef784428e9d

This makes the report fully auditable. Every important claim should point back to an explicit evidence_ref.

Claude Code harness: command, coordinator, specialists

The main entry point is a Claude Code slash command:

/investigate-incident namespace=si service=multi-system-processor symptom="readiness probe failures during recent rollout" since_minutes=60

The command does not directly perform the whole investigation. It routes the work to an incident-coordinator that manages state within a dynamic scratchpad:

runs/<investigation_id>/scratchpad/
  coordinator-brief.md
  k8s-evidence-collector.md
  prometheus-analyst.md
  log-analyst.md
  runbook-analyst.md
  incident-reporter.md

The coordinator delegates to specialist subagents with narrowed focus scopes:

• k8s-evidence-collector: Gathers current pod state, descriptions, events, logs, and resource usage via top.
• prometheus-analyst: Checks restarts, CPU, memory, latency, and error rates.
• log-analyst: Queries IBM Cloud Logs for historical errors and probe failures.
• runbook-analyst: Correlates known patterns and returns recommended safe checks.
• incident-reporter: Compiles the final evidence-grounded report.

The coordinator passes explicit context to each specialist. This avoids relying on hidden context inheritance and keeps each subagent lightweight and hyper-focused. Since operational investigations naturally split into distinct collection, analysis, matching, and synthesis phases, this architecture maps perfectly to the problem space.

Symptom-driven routing

The investigation is not a generic health check. It starts with a concrete, contextual symptom. For instance:

• OOM / Restarts / Crash Loop: Evaluates k8s_list_pods, pod descriptions, namespace events, and Prometheus restart metrics.
• Readiness / Liveness Failures: Looks at namespace events, pod definitions, and queries log patterns for probe failures.
• Latency / Errors: Polls p95 latency thresholds and HTTP error rates.

This strict routing prevents investigation drift. If a service is alerting on a readiness failure, unrelated application logs shouldn't capture the agent's attention until the core symptom's window is thoroughly accounted for.

Handling Observability Gaps Realistically

Metrics are extremely useful, but observability gaps are common. The project treats missing or unreachable Prometheus endpoints as an unknown variable, not as proof of optimal system health.

For example, the engine enforces these logical rules:

• Prometheus unreachable does not equal zero restarts.
• Missing metric does not equal zero application errors.
• No latency metric does not equal baseline performance.

The coordinator uses explicit validation tools like prom_ensure_connection. If the target metric system is unreachable, the query is explicitly labeled as unverified, and the final report calls out the metrics gap directly rather than turning missing data into a confident, incorrect conclusion.

If Prometheus is reachable, the metric query is retried. If not, the final report lists the metrics gap explicitly.

Safety hooks and report validation

The Claude Code harness includes global lifecycle hooks for strict safety auditing:

• PreToolUse: Intercepts and blocks unauthorized or unsafe shell commands.
• PostToolUse: Automatically records and logs all incoming/outgoing tool payloads.
• SubagentStart / SubagentStop: Traces and profiles subagent lifecycles.
• Stop: Enforces schema validation against the final output report.

The final report validator verifies that the generated document satisfies the structural schema, validating fields such as evidence_ref tracking, ruled-out elements, explicit unknowns, and a clear diagnostic verdict. We are not trusting the agent blindly—the harness sets up the rules, tracks execution, and audits the exact evidence output structural formatting.

What Claude Code made easier

Claude Code provided excellent foundational primitives for the orchestration layer:

1. Slash commands as a clean user entry point.
2. Coordinator and specialist subagent abstraction.
3. Explicit workflow routing rules.
4. Scratchpad isolation file discipline.

The subagent model was particularly useful. It allowed the workflow to separate evidence collection from metrics analysis, log analysis, runbook matching, and final reporting.

That separation made the final report easier to audit.

The value of the harness is that it turns “ask an AI to debug this” into a repeatable investigation process.

Harness portability: IBM Bob experiment

After building the Claude Code harness, I tested whether the same MCP-native app could run under another harness - IBM Bob.

I added a Bob-specific harness under:

.bob/
  mcp.json
  custom_modes.yaml
  skills/investigate-incident/SKILL.md
  rules-ops-investigator/

Bob reused the same MCP server and the same typed tools. The Bob workflow was skill-based and single-agent, while Claude Code used coordinator and specialist subagents. But the important lesson was that the core app did not need to be rewritten.

Design Principle Verified:
Keep durable business logic and core operational schemas within your MCP tools and application layers. Keep your terminal-specific presentation layers cleanly separated within .claude/ or .bob/ directories.

Conclusion

Claude Code can be far more than just an IDE coding assistant. When paired with structured Model Context Protocol (MCP) toolkits, rigorous evidence schemas, subagent separation, and lifecycle hooks, it transforms into an incredibly reliable runtime for complex SRE workflows. For production Kubernetes troubleshooting, this design moves the needle away from ad hoc, chat-based guesswork and into highly repeatable, secure, and fully auditable incident investigation automation. The code is available on my GitHub - github.com/Randhir123/claude-ops-investigator