Splitting a job
without losing it.
Running many agents at once is easy. Getting one coherent deliverable back out is the hard part, and almost all of the difficulty is at the seams.
Diagram: lead → briefs → waves of workers → merge gate → one result. A clean line drawing rather than a screenshot.
16/9Three decisions that did most of the work
The lead never does the work
It is tempting to let the planning agent also take the first task — it already has the context, and it is sitting idle. It is a mistake. The moment the lead starts working it stops holding the whole job in view, and the plan degrades into whatever the lead happened to be looking at. The lead plans, commissions, and reads results. Nothing else.
Check at merge, not at write
The first version checked whether a worker had strayed outside its brief at the moment it wrote a file. That produced constant false positives, because a worker legitimately touches shared things on the way to its own goal. Moving the check to merge fixed it: at merge you can see every worker’s output at once, which is the only point where "did this stay in scope" is actually answerable.
The merge gate rejecting work, with the reason shown.
4/3Six at a time, three hundred in total
The concurrency limit is not a performance ceiling we intend to lift. Past a handful of concurrent workers the provider starts rate-limiting, and a rate-limited worker fails in a way that looks like a bug rather than a queue. Six keeps the machine usable and the run alive. Three hundred per run is the size of job this is for.
The failure that taught us the most
A three-hour run produced three overlapping waves of workers writing the same filenames, eight refused handovers, and hours of wasted work. The cause was not in the swarm at all: the parent conversation had been compacted, forgot it had already commissioned the work, and ordered it again. The fix belongs at the boundary between conversation memory and job state — a lesson about where systems actually break rather than where you look first.
FAQ
Why not run more workers concurrently?
Provider rate limits. Past roughly six, failures stop being occasional and start being the normal case, and a rate-limited worker fails in a way that is hard to distinguish from a real error.
What happens to work that fails the merge check?
It goes back with the reason. It is not silently accepted, and it does not take the run down.
Does each worker see the whole conversation?
No. Each gets its own brief and its own context. That is most of why the approach works at all — a worker carrying everything is a worker carrying mostly noise.