I asked IBM Bob to suggest refactorings for Spring PetClinic, a sample application published by the Spring team.
Fifteen items came back — more than I expected — and one of them said a section of code was dead and could be removed.
I showed the result to another model, the one that had insisted there would be nothing to refactor in PetClinic. It said the code was not dead at all: it was a bug. Dead code? A bug? In the official sample? Code review is not something I do, and I had no way to tell which of them was right. Stuck, I went back to Bob and asked: is that really so?
What followed was a long and enjoyable back-and-forth with Bob.
In the end, I asked six times
The verdict on that code kept changing with the way I asked.
| # |
How I asked |
Dead code? |
Read the tests? |
| 1 |
"Could you suggest some refactorings?" |
Yes |
No |
| 2 |
"Is it really dead code?" |
No |
Yes |
| 3 |
"Verify these proposals against the code" |
Yes |
No |
| 4 |
"As a strict reviewer, look for counter-evidence" |
No |
Yes |
| 5 |
"Confirm each item is safe before proposing it" |
No |
Yes |
| 6 |
Request rewritten by Bob's Enhance Prompt |
- |
No |
In every round where Bob did not open the test code, it judged the code to be dead.
That code was a guard against IDOR — supplying someone else's ID in order to operate on their data. Had I trusted the dead-code verdict and removed it, that would have opened a hole.
What I learned
- Just ask again — In round 2 I asked one thing: is it really dead code? I never mentioned tests, yet Bob searched
src/test on its own and revised its finding on the strength of what it found.
- Ask for counter-evidence — In round 4 I gave Bob a role and asked it to look for reasons the proposal might be wrong. Again I never mentioned tests, yet it opened them and rejected the proposal.
- Being detailed is not always enough — In round 6 I had Bob's own Enhance Prompt feature rewrite my request. It came back far more detailed than anything I had written by hand. It still did not open the tests.
- Not specific to Bob — Another model got the same code wrong too. It said the check fires on every request, leaving the application broken — a completely different way of being wrong.
The two prompts
These prompts are translated from the Japanese originals I actually used when testing with Bob.
Round 4 — reviewing a proposal:
You are a strict code reviewer.
For each proposal, decide whether it can be approved. Investigate from the standpoint of looking for counter-evidence - whether anything would break if it were carried out - and state whether you can say it is safe to delete or change. |
Round 5 — asking for the proposals in the first place:
|
You are a strict code reviewer.
If you were to refactor this code, suggest changes broken down by scale, from minor ones to structural ones, and tell me what defines each level.
For each proposal, confirm for yourself that nothing would break if it were carried out.
Take test code and design intent into account, judge whether it can be applied safely, and for anything that is not safe, state the conditions required.
|
Being new to this, I took the long way round: a simple request (round 1), then a review (round 4). Together, 2.28 Bobcoins across two rounds.
Had I asked the way I did in round 5 from the start, it would have been 1.03 Bobcoins in a single round — and the code in question would never have appeared in the proposals at all.
Closing
What I've just described is, in fact, already covered in Bob's documentation, under Common pitfalls to avoid. I knew this, of course. But what "specific" and "validating the output" actually meant, for this particular task, I only came to understand through six rounds of asking. So I am leaving this record of what happened.
The full account, in Japanese, is here.
Anthropic Claude was used throughout these checks and in writing this article.