It was a Tuesday evening, and the agent had done everything right. Read the ticket, found the three files that mattered, wrote a change that did exactly what the description asked for. Typecheck clean, tests green. Then git push - and an error in red. The token it was running under had read-only scope. The branch existed inside a container the pipeline tore down twenty seconds later.
The work was done. It was worthless anyway.
We tell that story fairly often now, because it maps the gap precisely. What separates an agent that impresses in a demo from one you can point at a repository and walk away from is not more intelligence. It is infrastructure.
In a demo, you are the transport layer
Give a model a task in a chat window, watch it return a clean solution, and something happens next that nobody counts. You copy the code into your editor. You put the file in the right place. You notice a missing import and add it without thinking. You run the tests. You commit.
Six steps, done so automatically that they never enter your assessment of the model. The impression you walk away with is “this thing can code”. What you actually watched was a suggestion that a human turned into software.
Take the human out and those six steps become visible. Every one of them can fail, and none of them is code generation. That is the real job: isolation, verification, a clean handover point.
Isolation: somewhere the agent is allowed to be wrong
An agent needs its own workspace - a worktree, a branch, a container. Not primarily because it is dangerous, but because a half-finished change in a shared checkout blocks everyone else, and because two agents cannot work in parallel if they are editing the same files.
Isolation is also what makes a failed attempt cheap. If an agent writes a change in its own worktree and the change does not hold up, you throw the worktree away. If it does the same thing in the main checkout, somebody cleans up after it. That difference decides whether you are willing to run the agent at all when you are not sure it will succeed.
Verification: against a real environment, not against a diff
This is where it gets interesting, and where we see most setups come apart.
The obvious check is that a human reads the diff. Looks plausible, gets merged. The problem with that is structural - a diff is the author’s statement of intent, not the system’s statement of fact. It tells you what somebody meant to change. It does not tell you what happens afterwards.
We have collected enough examples that the pattern is easy to recognise. A new route sits in exactly the right place in the source but never makes it into the build because the file suffix is wrong - the diff reads perfectly, the URL answers 404. A translation exists as a file, but the language switcher never links to it because one frontmatter field is missing. A redirect lives in middleware that a static build never executes. In all three cases the diff is correct in the sense that it contains what it is supposed to contain. And in all three cases the change does not exist for the visitor.
So “verified” means something specific to us: somebody asked the running system. Not the source, the system. Request the URL and look at the status code. Read the rendered HTML and check that the link is really on the page. Run the build and confirm the file landed in dist/.
That is a different question from review. Review asks whether this is good code. Verification asks whether it exists in the world. You need both, and one does not substitute for the other. A card can be closed as done twice and still not be done, because both times only the first question got asked.
The case that taught us the most is the one that is already deployed. The change is merged, the pipeline is green, the feature is live. What is left to verify? Not “the diff looked fine” - that was true before the merge. It means the deployed instance shows the behaviour the ticket described. Miss that distinction and you end up fixing a state of the code while the bug on the server carries on exactly as before.
The handover point
At the end of the loop there is a human, and the transition to them is a deliverable in its own right.
An agent that leaves a pull request titled “Fix” with a body reading “done” has not saved anyone time. It has moved the time - from writing to reconstructing, and reconstructing is the more expensive activity. What we ask for instead: what changed and why, tied back to the ticket. The test output showing the change is covered. And a paragraph on what a reviewer should look at closely, including whatever did not work.
That last part matters most and is the part agents volunteer least. An agent that could not finish some of the task and says so plainly is useful. One that quietly leaves it out costs you more than not running it at all, because from then on you cannot take any of its reports at face value.
Where the work is now
Code generation is the part that is largely solved. It is good, it keeps getting better, and it is no longer a differentiator.
The rest is. An agent that works in isolation, verifies against a real environment, and hands over cleanly to a person delivers software. An agent without that loop delivers suggestions - occasionally excellent ones, and still only suggestions. The difference is not the model. It is the loop around it, and that part you have to build.
