A break worth adding to the simulator: let the agent edit the check itself, by skipping a test or loosening an assertion, and show the loop "succeed". That's the failure people miss most, because the judge step passes honestly against a weaker check. Marking test and CI files as read only during the loop, and showing that in the simulator, would make the point in one click.
That's a really good suggestion! Letting the agent modify its own tests is definitely something worth demonstrating. I'll look into adding that scenario to the simulator
This is a useful way to think about reliable AI automation. At IT Path Solutions, we believe an effective agentic workflow needs more than a plan-build-repeat cycle; it also needs independent validation, stable acceptance criteria, and risk-based approval boundaries. These safeguards help ensure that agents don't just produce working-looking results but deliver outcomes that genuinely meet business and technical requirements.
Thanks Glen! Approval boundaries are especially important when agents start making changes to production systems. Definitely something worth exploring further
Absolutely! Production environments make these boundaries even more critical. I'd also distinguish between actions an agent can safely execute automatically and actions that require human authorization, especially when changes affect live services, infrastructure, or customer data. Having clear permission levels and rollback mechanisms can make agentic workflows much safer to operate.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The recorded run where the agent claimed "All 8 tests pass" while only 6 actually ran is the detail that stayed with me. It's the perfect example of a loop stopping on a check that doesn't measure the goal at all — the agent wasn't lying, it just never looked. That's exactly why I keep a separate judge step around, even when it costs more tokens.
Yeah that was a pretty interesting finding! The agent was confident all 8 tests passed without even checking how many actually ran. A separate check of the test output would have caught that pretty quickly
The judge step is where a loop either pays back or quietly multiplies its cost. If it reads the whole transcript on each pass, every extra iteration costs more than its predecessor. Does your judge see the full transcript, or just the diff and the test output?
Good point! The simulator uses a scripted flow so there's no actual token cost here. For a real implementation I'd probably give the judge the requirements, diff and test results rather than the full conversation history
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the 200 versus 201 case with the judge turned off is the one i keep seeing in real harnesses. tests go green, the builder says done, and nobody compared the output to the spec. same story with the two tests under the main guard that unittest never collected.
what helped us was making the judge read only the spec and the diff, never the builder summary, so it cannot inherit the builder story. tokens went up but fewer bugs escaped.
in the real runs you recorded, did the judge ever approve something the spec simply did not cover?
That's an interesting approach to keeping the judge independent from the builder's assumptions!
The recorded example focused on the test count issue so I don't have a confirmed case of the judge approving something outside the spec. Would be interesting to test that in a follow-up though
I am a professional DevOps Engineer with a demonstrated history of working in the internet industry. I am an avid Linux lover and supporter of the open-source movement philosophy.
@devopsdaily Right, and that is where the cost lives: the judge is billed on whatever you hand it, on every pass. Requirements, diff and test output stay bounded by the change itself, not by how long the loop has been running.
One point worth stressing is that the harness, not the model, owns the stop condition and the retry budget, which is exactly where most runaway loops hide. In a desktop app we hit the same boundary: a renderer-side agent that keeps re-issuing an IPC call after the main process already failed needs an explicit max-iteration counter and a terminal-state check, not just a model that thinks it should stop. Splitting plan/build/judge into separate evaluator and optimizer calls also makes it far easier to log each loop iteration and catch the case where the judge keeps rubber-stamping a failing build.
Self-taught full-stack engineer. Founder of ML Systems, a Rhode Island construction tech company. Shipped a React Native app to both stores plus the Next.js/tRPC/Postgres platform behind it.
"Tests green is not the same as goal met" is the caption I'd frame. A wall can pass the nailing inspection and still be in the wrong place on the plan. Having someone other than the builder check against the spec is why the inspector doesn't work for the framer. What done means is the part you design, and I call the person who writes it a Language Modeler.
We're a place where coders share, stay up-to-date and grow their careers.
A break worth adding to the simulator: let the agent edit the check itself, by skipping a test or loosening an assertion, and show the loop "succeed". That's the failure people miss most, because the judge step passes honestly against a weaker check. Marking test and CI files as read only during the loop, and showing that in the simulator, would make the point in one click.
That's a really good suggestion! Letting the agent modify its own tests is definitely something worth demonstrating. I'll look into adding that scenario to the simulator
Looking forward to it. An agent quietly editing its own test to get green is the scenario that makes the case for keeping checks outside its reach.
This is a useful way to think about reliable AI automation. At IT Path Solutions, we believe an effective agentic workflow needs more than a plan-build-repeat cycle; it also needs independent validation, stable acceptance criteria, and risk-based approval boundaries. These safeguards help ensure that agents don't just produce working-looking results but deliver outcomes that genuinely meet business and technical requirements.
Thanks Glen! Approval boundaries are especially important when agents start making changes to production systems. Definitely something worth exploring further
Absolutely! Production environments make these boundaries even more critical. I'd also distinguish between actions an agent can safely execute automatically and actions that require human authorization, especially when changes affect live services, infrastructure, or customer data. Having clear permission levels and rollback mechanisms can make agentic workflows much safer to operate.
The recorded run where the agent claimed "All 8 tests pass" while only 6 actually ran is the detail that stayed with me. It's the perfect example of a loop stopping on a check that doesn't measure the goal at all — the agent wasn't lying, it just never looked. That's exactly why I keep a separate judge step around, even when it costs more tokens.
Yeah that was a pretty interesting finding! The agent was confident all 8 tests passed without even checking how many actually ran. A separate check of the test output would have caught that pretty quickly
The judge step is where a loop either pays back or quietly multiplies its cost. If it reads the whole transcript on each pass, every extra iteration costs more than its predecessor. Does your judge see the full transcript, or just the diff and the test output?
Good point! The simulator uses a scripted flow so there's no actual token cost here. For a real implementation I'd probably give the judge the requirements, diff and test results rather than the full conversation history
the 200 versus 201 case with the judge turned off is the one i keep seeing in real harnesses. tests go green, the builder says done, and nobody compared the output to the spec. same story with the two tests under the main guard that unittest never collected.
what helped us was making the judge read only the spec and the diff, never the builder summary, so it cannot inherit the builder story. tokens went up but fewer bugs escaped.
in the real runs you recorded, did the judge ever approve something the spec simply did not cover?
That's an interesting approach to keeping the judge independent from the builder's assumptions!
The recorded example focused on the test count issue so I don't have a confirmed case of the judge approving something outside the spec. Would be interesting to test that in a follow-up though
Cool simulator
Thanks Bobby! Glad you liked it
@devopsdaily Right, and that is where the cost lives: the judge is billed on whatever you hand it, on every pass. Requirements, diff and test output stay bounded by the change itself, not by how long the loop has been running.
One point worth stressing is that the harness, not the model, owns the stop condition and the retry budget, which is exactly where most runaway loops hide. In a desktop app we hit the same boundary: a renderer-side agent that keeps re-issuing an IPC call after the main process already failed needs an explicit max-iteration counter and a terminal-state check, not just a model that thinks it should stop. Splitting plan/build/judge into separate evaluator and optimizer calls also makes it far easier to log each loop iteration and catch the case where the judge keeps rubber-stamping a failing build.
"Tests green is not the same as goal met" is the caption I'd frame. A wall can pass the nailing inspection and still be in the wrong place on the plan. Having someone other than the builder check against the spec is why the inspector doesn't work for the framer. What done means is the part you design, and I call the person who writes it a Language Modeler.