A follow-up to my earlier post on building a Claude Code harness for Kubernetes incident investigation, this time built entirely on IBM Bob. Code: github.com/Randhir123/claude-ops-investigator.

The problem: two separate, manual jobs

When something breaks in one of our Kubernetes services, an on-call engineer's job usually looks the same every time: check the pods, check Prometheus, check the logs, maybe check a runbook, and stitch all of that together into "here's what's probably wrong." That alone can eat half a day of tab-switching.

And if the root cause turns out to be an actual bug in the code — not a config issue, not a resource limit, an honest-to-god defect — there's a second job that starts from scratch: find the file, understand what's happening, write a fix, open a pull request. Before this project, that whole cycle — investigate, then locate and draft a fix — routinely took us a day or two per incident.

This post is about closing both of those gaps with IBM Bob — not by replacing the engineer, but by giving Bob a scoped, safe way to do the tedious first 90% of both jobs.

Part 1: Bob investigates, like a team of specialists

The first piece of this project (covered in more depth in my earlier post) turns a single Bob session into something closer to a small team. Instead of one general-purpose agent poking around a cluster with raw kubectl, Bob runs as an orchestrator that delegates to five narrowly scoped custom modes, each with its own tightly limited toolset:

• k8s-evidence-collector — pod status, live logs, recent events
• prometheus-analyst — restart counts, CPU/memory, error rate, latency
• log-analyst — historical log search across restarts and deployments
• runbook-analyst — matches the symptom against known incident patterns
• incident-reporter — never touches a tool at all; just synthesizes everyone else's findings into one structured report

None of these modes talk to Kubernetes or Prometheus directly. They all go through a small MCP server exposing about 18 narrow, read-only tools — no raw shell, no generic kubectl — and every finding a specialist reports back carries an evidence_ref pointing at the full raw data, so nothing in the final report is a guess. One hard rule we added: if Prometheus itself is unreachable, the investigation stops immediately rather than quietly treating "no metrics" as "everything's fine." A gap is a gap, never a zero.

The whole thing runs from one command:

/investigate-incident namespace=si service=event-data symptom="intermittent dropped events" since_minutes=60

Part 2: when the cause is a real bug, propose the fix

The investigation piece was already useful on its own. The part we built for this challenge goes one step further: when the report traces the cause to a specific, named spot in application code — not a resource limit, not a probe timeout, an actual bug — Bob can locate that code, write a minimal fix, and open a draft pull request. Fully on its own, with no human approval step in the middle.

That last sentence is exactly the kind of thing that should make you nervous, so here's how we kept it from being reckless.

It's a separate command, on purpose

/investigate-incident never changes behavior based on what it finds — it always just produces a report. A brand-new command, /propose-fix, is the only thing that can trigger a code change, and you have to type it deliberately. A routine investigation can never quietly turn into a code push just because Bob happened to notice something code-shaped.

It only fires on a real code-level cause

Before doing anything else, Bob re-checks: does the report cite a specific exception, a specific file, a specific function? If the incident is really about a memory limit or a flaky readiness probe — which is most of the time — Bob does nothing further. That's not a failure; it's the correct outcome. We didn't want "vaguely code-shaped" to be good enough.

It only ever works in your existing checkout, never clones one

Bob doesn't go create a fresh clone somewhere. It works directly in the service's own local checkout that's already on your machine, and the very first thing it does is check that checkout is clean. If there's any uncommitted change to a tracked file, it stops and tells you — it will never risk mixing its own branch with work you haven't committed yet. (We did have to teach it the difference between "actual uncommitted work" and "the usual pile of untracked IDE files and log folders every real repo seems to accumulate" — more on that below.)

The PR is always a draft, and always says what it is

Every pull request Bob opens starts with the same bolded line: this is an AI-proposed fix from an automated incident investigation, it has not been reviewed. It includes the evidence the fix is based on and a human-review checklist. And it's opened as a draft, every time — autonomy up to the point of "here's a PR for a human to look at," never past it.

Watching it actually work

We tested this against a real bug in one of our services, event-data. The investigation traced a pattern of dropped Kafka messages to a specific catch block in EventProcessor.java that logged the error but never incremented any metric — meaning the drops were invisible to monitoring, only discoverable by grepping logs.

We ran /propose-fix against that report. Bob:

1. Confirmed the cause was genuinely code-level, not infrastructure
2. Checked out the real local copy of the repo, confirmed the working tree was clean of anything except normal untracked cruft
3. Found the actual file and read the surrounding code
4. Verified a real metrics counter it could hook into, rather than inventing a plausible-sounding method name
5. Created a branch, pushed it, and opened a real draft pull request

One file changed. One commit. A branch named after the investigation. A draft PR waiting for a human. Exactly the scope we designed for.

What we ran into (and fixed, with Bob's help)

Two real problems came up while building this, both worth mentioning because they're the kind of thing that only shows up when you actually run something, not when you just design it on paper:

• Bob Shell's tools are scoped to the project you launch it from. Reaching a different local checkout on the same machine needed one extra setting (includeDirectories), not something we'd have guessed without hitting the error first. The first time fix-proposer tried to cd into event-data's checkout, it failed with:
  

  Error: Directory /Users/.../eclipse-workspaces/tss-split-1/event-data
    is not within any of the registered workspace directories.

  That's a different gate than "trusted folders" (which only controls whether .bob/ project config loads for the directory Bob was launched from) — it's a separate setting that controls what the command tool is allowed to reach at all. The fix was adding the source repo's path to context.includeDirectories in Bob Shell's settings:

  // ~/.bob/settings.json
    {
      "context": {
        "includeDirectories": [
          "/Users/.../eclipse-workspaces/tss-split-1/event-data",
          "/Users/.../eclipse-workspaces/tss-split-1/time-series-query",
          "/Users/.../eclipse-workspaces/tss-split-1/multi-system-processor"
        ]
      }
    }

  One catch: Bob Shell reads custom modes and settings once at session startup, not hot-reloaded — so this only took effect after a full restart, the same class of gotcha we'd already hit once before with .bobignore.
• Our first "is this checkout safe to touch" check was too strict — it flagged completely harmless untracked files (an IDE project file, some .gitignore stubs, a log folder) as if they were uncommitted work, which would have blocked every real run. We tightened it to only care about changes to files git is actually tracking.

Both were caught by actually running the thing against a real checkout before trusting it — not something you'd find by reading the rules file.

Conclusion

Investigation and fix-proposal are two clearly separate, clearly scoped capabilities now, each safe on its own terms: one always read-only, the other autonomous but fenced in by a narrow trigger, a draft-only PR, and a clean-tree check that refuses to touch anything it doesn't fully understand. Neither replaces judgment — they just mean the judgment a human applies starts from a report and a proposed diff, instead of a blank terminal.