A few months ago I wrote about building a software factory. This is the sequel: what the factory looks like now, what we got wrong, and the part where the pipeline started reviewing its own builder.
Every team using AI agents goes through the same three phases. You just might not know you're in phase one yet.
phase one: you are the conveyor belt
Our first workflow was a set of steps with a human between every pair of them. Discuss the idea. Approve. Write the ticket. Approve. Plan. Approve. Implement. Review. Approve. Test. Approve. Each step had a skill, which is just a reusable instruction file the agent follows, and each skill was genuinely good.
It worked. It also had a ceiling, and the ceiling was me. Every arrow in that diagram passes through a person. You become the conveyor belt between stations. The agents were fast. I was not.
Here's the thing we only saw later: the steps were never the valuable part. The checks were. The evidence rules, the reviews, the tests. Those are what kept the output honest. The approvals in between were mostly me ferrying context from one station to the next.
phase two: gates instead of approvals
My teammate Tyler rebuilt the workflow so it runs itself. We call it Orchestra. The idea is simple to say and took months to earn: replace human approvals in the middle with automated gates, and keep the human at the two spots where judgment actually matters. Deciding what to build. And reviewing the finished pull request.
The flow looks like this. You talk through an idea with the agent (we wrote about how to have that conversation well). The decisions get written to a log file, so nothing lives only in chat history. When the idea has a shape, you capture it as a work item: what you want, why, and how we'll know it's done, written as checks a machine can verify.
Then comes my favorite part. Before the item is accepted, an agent we named Socrates argues against it. Its whole job is to attack the premise. Do we need this at all? Is this the root cause or a symptom? Is there a simpler way? If your answers hold up, they get attached to the ticket, so six weeks later you can see why the work exists. If they don't hold up, you just saved days of building the wrong thing.
After the gate, one command runs the whole build. Plan, implement, verify, open the PR. No human in the middle. The safety comes from rules baked into each step:
1. The plan can't claim anything about the codebase without pointing at the exact file and line that proves it. If the claim is "this function is never called," the plan has to show the search that came up empty. This one rule kills most hallucinated plans.
2. Every review runs twice, in parallel, using two different AI models from two different companies. They have different blind spots. A problem has to sneak past both to ship. This is the same idea as giving one chef six hats: fresh eyes, different role, same kitchen. The work only proceeds when neither reviewer has a blocking finding.
3. Nothing counts as done because the agent says so. Every acceptance check needs quoted output: the test run, the command result, a screenshot of the app doing the thing.
4. When reviewers find problems after the PR opens, fixes get pushed to the same PR, and the loop repeats until it's clean. The PR isn't the finish line. It's just the first artifact worth showing you.
And when a result still falls short? There's a step for that too. A postmortem that asks why the system produced a miss, and changes one rule, one template, one gate. The code fix is the symptom. The system fix is the point. That's how the factory gets better at making things instead of just making things.
the day it reviewed its own builder
Here's the moment this stopped being theory for me. We upgraded the pipeline using the pipeline. I wrote a plan for the upgrade, and the plan included a claim about the codebase with a search as evidence. Both reviewers, independently, ran my search and caught that it didn't return what I said it did.
The evidence rule I had added that same morning flagged its own author before lunch. That's when you know a check is real: it doesn't care who it embarrasses.
phase three: the factory finds its own work
Phase two still starts with a person typing an idea. Phase three is where the inputs stop waiting for you. Error trackers already know what's crashing. Analytics already know where users get stuck. Security scanners already know which packages are stale. Today that information sits in dashboards, waiting for someone to look.
So the next stage is scheduled agents that read those sources on a timer, turn real signals into work items, and feed them through the same gates as everything else. The Socratic gate matters even more here: when nobody typed the idea, something still has to ask whether it's worth building. Wake up to three small PRs, each fixing something real, each with evidence attached. Shipped while you sleep.
We're building this part now. The diagram at the top is honest about that: it's the only section drawn in future tense.
what i'd tell you if you're starting
Don't start by automating. Start by writing down your checks. What would convince you a change is correct? What evidence would you want to see? Make the agent produce that evidence while you're still approving every step. That's phase one, and it's not wasted work. Every check you write there becomes a gate later.
Then remove yourself from one spot at a time. Let reviews run without you first. Then verification. Then planning. Keep the two edges for yourself: deciding what's worth building, and looking at what got built. Those were never the bottleneck. They were the job.
The tooling underneath matters too. Agents running in parallel need isolated copies of the codebase so they don't overwrite each other, and you need one place to see them all. That's the problem Pane exists to solve, and this workflow is why we built it. If you want the earlier chapters, start with how we use Claude Code and agent loops.