Splitting "what the agents decide" from "what I still decide" is the most useful framing I've read on this. One question from our own setup: when a human answers an open question at the spec stage, where does that answer live for the next feature? Ours used to vanish with the PR. Disclosure: I build brethof-brain, which keeps answers like that as dated decisions and hands the relevant ones to the agent on the next prompt that touches them.
Lead Frontend & AI Engineer. I build agentic workflows on Claude Code and measure if they actually work: 87% lower LLM cost, 86% of agent code passing typecheck first try. 15+ yrs building web.
Thanks! In our setup they live in two places. Decisions that should outlive the feature go into the project instructions file the agent reads on every session, with a date and a short history: what we assumed, what broke, how we checked it. Everything else stays in a per-branch doc next to the spec. The weak spot is the step between them: promoting a decision from the branch doc to the shared file is still manual, and sometimes it doesn’t happen. How does brethof-brain decide which past decisions are relevant to a prompt?
Your weak spot is the part we stopped doing by hand. Nobody promotes a decision: records are curated from the conversations themselves, so the branch decision is in the memory the moment someone says it. Then with each prompt, before the agent replies, the few records that bear on it are put in front of it whole, plus any rule it touches. The why behind each one, dated, comes up when the agent searches.
Lead Frontend & AI Engineer. I build agentic workflows on Claude Code and measure if they actually work: 87% lower LLM cost, 86% of agent code passing typecheck first try. 15+ yrs building web.
That answers it, thanks. Capturing at the moment it’s said removes the step that kept failing for us. What I’d want to keep is the split the pipeline already makes. A human’s answer lands under “Decisions”, and anything the backend hasn’t confirmed is marked ASSUMED until someone checks it. Our worst regression was an assumption stated confidently in a session and treated as fact. If records come straight from conversations, does a record know whether it was decided, assumed or verified? And when a later conversation reverses one, does the old record get superseded, or do both reach the agent?
Splitting the pipeline by who decides, rather than by which step comes next, is the useful framing here. The scout that can only read and must list gaps instead of guessing is a nice detail, and so is marking anything the backend has not confirmed as ASSUMED. Your note that the worst regression came from an assumption stated confidently and treated as fact is a lesson most teams only learn once.
I also agree with stopping after one red step. A second failure usually means the plan was wrong, and paying for more rounds on a wrong plan is expensive in both tokens and review time.
At Ascendra Ventures we scope client builds the same way at a higher level: the acceptance criteria are written and agreed before code, and each one has a test that has to fail before the fix exists. It makes the end of the job something both sides can check.
When a decision gets promoted from the branch doc to the shared instructions file, who reviews that promotion, you or the agent that proposed it?
Tokenade is my project, and I hit the same split on my own agent pipeline. A rule in a prompt can't be tested, so the agent drifts without anyone seeing it. What can be checked (a count, a format, a missing field) moved into code with a test. The model keeps the judgment: which angle, how to write it.
Lead Frontend & AI Engineer. I build agentic workflows on Claude Code and measure if they actually work: 87% lower LLM cost, 86% of agent code passing typecheck first try. 15+ yrs building web.
Same lesson here. Our rule “never commit without passing tests” lived in the instructions file and got skipped now and then. It works since it moved into a pre-commit hook. For what can’t be a check, I still measure drift from the transcripts: how often the rule held, not whether the prompt says it. Which rule was the hardest to move into code?
@diegoprieri where do you draw the line when the test was written by the author of the fix? That's the case where the countable edges pass and the judgment is least trustworthy.
Lead Frontend & AI Engineer. I build agentic workflows on Claude Code and measure if they actually work: 87% lower LLM cost, 86% of agent code passing typecheck first try. 15+ yrs building web.
That’s the case I trust least too. Three things narrow it, none of them close it. The test comes first and has to fail before the fix exists, so it can’t be shaped around the fix. What it checks comes from acceptance criteria another stage wrote, before any code, and each test has to name the criterion it covers. And a separate reviewer agent, with no access to the fixer’s reasoning, reads the diff and asks whether the test checks the criterion or just the implementation. What’s left is a test that faithfully checks a wrong criterion. No agent catches that; it’s why the PR is a draft and a human reads the tests before the code.
@diegoprieri The hardest ones look countable but need a judgment call, like what counts as a real source. A count can't settle that, so it stays with the model, and code checks only what surrounds it.
Lead Frontend & AI Engineer. I build agentic workflows on Claude Code and measure if they actually work: 87% lower LLM cost, 86% of agent code passing typecheck first try. 15+ yrs building web.
Same pattern here. “Tests pass” is countable. “This test checks the right thing” isn’t, so that stays with review. What helped was checking the edges code can see: the test exists, it ran, it failed before the fix. The judgment in the middle stays human, or with the model.
We're a place where coders share, stay up-to-date and grow their careers.
Splitting "what the agents decide" from "what I still decide" is the most useful framing I've read on this. One question from our own setup: when a human answers an open question at the spec stage, where does that answer live for the next feature? Ours used to vanish with the PR. Disclosure: I build brethof-brain, which keeps answers like that as dated decisions and hands the relevant ones to the agent on the next prompt that touches them.
Thanks! In our setup they live in two places. Decisions that should outlive the feature go into the project instructions file the agent reads on every session, with a date and a short history: what we assumed, what broke, how we checked it. Everything else stays in a per-branch doc next to the spec. The weak spot is the step between them: promoting a decision from the branch doc to the shared file is still manual, and sometimes it doesn’t happen. How does brethof-brain decide which past decisions are relevant to a prompt?
Your weak spot is the part we stopped doing by hand. Nobody promotes a decision: records are curated from the conversations themselves, so the branch decision is in the memory the moment someone says it. Then with each prompt, before the agent replies, the few records that bear on it are put in front of it whole, plus any rule it touches. The why behind each one, dated, comes up when the agent searches.
That answers it, thanks. Capturing at the moment it’s said removes the step that kept failing for us. What I’d want to keep is the split the pipeline already makes. A human’s answer lands under “Decisions”, and anything the backend hasn’t confirmed is marked ASSUMED until someone checks it. Our worst regression was an assumption stated confidently in a session and treated as fact. If records come straight from conversations, does a record know whether it was decided, assumed or verified? And when a later conversation reverses one, does the old record get superseded, or do both reach the agent?
Splitting the pipeline by who decides, rather than by which step comes next, is the useful framing here. The scout that can only read and must list gaps instead of guessing is a nice detail, and so is marking anything the backend has not confirmed as ASSUMED. Your note that the worst regression came from an assumption stated confidently and treated as fact is a lesson most teams only learn once.
I also agree with stopping after one red step. A second failure usually means the plan was wrong, and paying for more rounds on a wrong plan is expensive in both tokens and review time.
At Ascendra Ventures we scope client builds the same way at a higher level: the acceptance criteria are written and agreed before code, and each one has a test that has to fail before the fix exists. It makes the end of the job something both sides can check.
When a decision gets promoted from the branch doc to the shared instructions file, who reviews that promotion, you or the agent that proposed it?
Tokenade is my project, and I hit the same split on my own agent pipeline. A rule in a prompt can't be tested, so the agent drifts without anyone seeing it. What can be checked (a count, a format, a missing field) moved into code with a test. The model keeps the judgment: which angle, how to write it.
Same lesson here. Our rule “never commit without passing tests” lived in the instructions file and got skipped now and then. It works since it moved into a pre-commit hook. For what can’t be a check, I still measure drift from the transcripts: how often the rule held, not whether the prompt says it. Which rule was the hardest to move into code?
@diegoprieri where do you draw the line when the test was written by the author of the fix? That's the case where the countable edges pass and the judgment is least trustworthy.
That’s the case I trust least too. Three things narrow it, none of them close it. The test comes first and has to fail before the fix exists, so it can’t be shaped around the fix. What it checks comes from acceptance criteria another stage wrote, before any code, and each test has to name the criterion it covers. And a separate reviewer agent, with no access to the fixer’s reasoning, reads the diff and asks whether the test checks the criterion or just the implementation. What’s left is a test that faithfully checks a wrong criterion. No agent catches that; it’s why the PR is a draft and a human reads the tests before the code.
@diegoprieri The hardest ones look countable but need a judgment call, like what counts as a real source. A count can't settle that, so it stays with the model, and code checks only what surrounds it.
Same pattern here. “Tests pass” is countable. “This test checks the right thing” isn’t, so that stays with review. What helped was checking the edges code can see: the test exists, it ran, it failed before the fix. The judgment in the middle stays human, or with the model.