Loop
A bug starts with a failing test
The agent turns a bug into a test that fails for the bug's reason, changes the code until it passes, and runs the whole suite.
- Reads
- Test runner
- When
- On every bug fix, before the code is changed and again after.
- Failure
- No test reproduces the bug, the test passes before the fix, the original reproduction still fails after it, or another test breaks.
- Next
- The agent rewrites a test that does not fail before the fix, and changes the code, never the existing tests, until all of it is green. When no test at a real seam can catch the bug, it says so in the task and shows a person the reproduction instead.
No illustration yet
catalog/loops/bug-failing-test-fix/card.svg The loop
- The agent reproduces the bug on its reported symptom, and shrinks the reproduction until every part of it is needed. It finds the cause before it changes anything.
- It picks the seam: the place where a test exercises the bug the way it happens at the call site. A test at a shallower seam passes while the bug stays.
- It turns the reproduction into a test at that seam, and watches it fail for the bug's reason. A test that passes before the fix does not reproduce the bug, and is rewritten before the code is touched.
- It changes the code, not the test, until the test passes.
- It runs the original reproduction again: shrinking it can drop a second cause, and then the small test passes while the reported bug remains.
- It runs the whole suite. A fix that turns the new test green and another one red is not done.
- The test stays in the suite, and the commit message names the cause that turned out to be right.
This is red, then green, applied to a bug.
Why
A fix with no test that reproduced the bug is a guess: nothing shows that the bug was there, or that the change removed it. A test that fails before the fix and passes after it shows both, and it stays in the suite, so the same bug cannot come back unnoticed.
Set up
- The instruction, where the agent reads its instructions: write a test that reproduces the bug and fails, then fix the code; run the original reproduction and the whole suite before saying it is done.
- The other tests are not the agent's to change. During a fix, a hook refuses its writes to existing tests; only the new test is its own.
- When no automated test can catch the bug (it is flaky, depends on the environment, or shows only in production), the agent says so in the task, reproduces it with a script, and the proof moves to confirming the release.
- Wiring. The agent runs the suite and the reproduction in its session, and a hook before the turn ends keeps it working while either is red.
Built from catalog/loops/bug-failing-test-fix/card.md.