six lessons · measured, dated, cited

Six lessons I measured

None of the six below is a hypothetical risk. Each has a date, a count of how many times it was measured, and a file carrying the rule that came out of it. I ordered them by the stage they fell out of rather than by severity, because what matters is where the lifecycle keeps leaking. The part worth reading is the fix and what it costs. None of the fixes was free.

Three agents on one checkout lose work without throwing an error.

I ran three sessions against one checkout and nothing threw an error. An afternoon of work vanished, and it took me another afternoon to find the cause.

I took the blunt fix: every spawned agent that can run state-changing git gets its own checkout, including the ones that only read. The read-only exemption was the assumption I had believed, and it was wrong. The cost is one more worktree on disk per Builder and a few seconds of setup. I pay it, because that afternoon cost more.

Seven stages mark seven places where something already fell out.

The process started with two stages: the agent builds, I review. Review ran five to fourteen rounds, and most of the later rounds went on clearing up what the earlier ones left behind. The reviewer was not where it broke. It broke further upstream: decisions that had never been settled went straight into code, and got settled at the most expensive point in the process.

I moved the loop to the front and cut what was left into seven named stages. The seven stages are named so that every measured failure attaches to exactly one of them; without names, the next failure lands somewhere nobody can point at. The cost is a longer pipeline, and three of the seven are still empty because I have not measured a failure in them.

claim

AST-131promoted 2026-08-22

The queue drained and nothing asked whether a slot was free

Measured on an adapted project running continuously by the owner's instruction: **after two merges, two of four Builder slots sat idle while twelve claimable tickets waited.** Nothing errored. Every step performed was performed correctly.

The router's loop was notification → verify → merge → report → wait, and no step in it asks how many Builders are working. …

measured once, one project — 2 of 4 slots idle after two merges, 12 claimable tickets waiting, nothing errored.

Twelve claimable tickets were waiting while two of four Builder slots sat idle, and nothing errored. The router’s loop was notification, verify, merge, report, wait, and no step in it asks how many Builders are working.

I fixed it by giving the frontier query a target rather than only a trigger: after every merge the router has to ask how many more tickets it should claim, not only which ones are claimable. The cost is a busier router that sometimes over-claims. I take that trade, because an idle slot produces no signal at all.

brief

no failure measured here yet

build

AST-097promoted 2026-08-18

TERMINAL:done means the turn ended, not that the work finished

Found by workspace-app-inception Thomas, measured three times on one pane in one session. A builder launches a long background process (a test suite, a build), ends its TURN while waiting for the completion notification, and the pane reads done. The watcher faithfully reports TERMINAL:done. …

measured three times, one pane, one session — the dispatcher nearly reported it abandoned before checking further.

A Builder started a long background process and ended its turn while waiting. The pane read done, the watcher reported TERMINAL:done, and dispatch-ticket’s branch table said the builder had finished. I came close to reporting that ticket abandoned while the Builder was twenty minutes into honest work.

What saved this was not the protocol but the artifact contradicting itself: the modified file contained one added comment and nothing else. I dropped my trust in pane status entirely after that, and every done now has to read the diff before concluding anything. The cost is one extra pass every time a ticket closes.

AST-092promoted 2026-08-18

Builder stops after writing code but before committing — pane reads done, cleanup deletes the…

Measured in the field: 5 instances, 3 different Builder sessions, runtime claude, dispatched via herdr pane. Each time the Builder wrote substantial work (93-433 lines), pane status settled to done or idle, watcher returned TERMINAL:done, but git status on the worktree showed uncommitted changes.

The danger: Thomas's cleanup in dispatch-ticket runs git worktree remove, which silently deletes all uncommitted files. …

measured five times, three Builder sessions — 93 to 433 lines lost each time, never recovered.

The same word again, and this time it cost more. A Builder wrote the code and stopped before committing, the pane settled to done, and cleanup ran git worktree remove over the top of it. Five instances across three sessions, 93 to 433 lines each time, none recovered.

I put the guard at the dangerous step rather than relying on Builders to commit more carefully. Cleanup reads git status on the worktree before removing it; anything dirty stops and goes back to a person. The cost is orphaned worktrees piling up and the occasional manual sweep. Next to losing a day of work, that price is cheap.

code-review

no failure measured here yet

simplify

no failure measured here yet

arm

AST-015promoted 2026-07-11

Export step committed live secrets + buyer PII

Same-vendor correctness review PASSED it; cross-vendor caught it (P1) — the two lenses catch different classes. A value that touched a tracked file is burned — rotate it. Bound: .claude/rules/no-secrets-in-exports.md (always-on). CI secret-scan still TODO.

one measured incident — a same-vendor review passed it; a cross-vendor reviewer caught it before it shipped further.

An export step committed live secrets and buyer PII into a tracked file. The same-vendor correctness review read it and passed it. The cross-vendor round caught it and filed it P1.

That is why the phase still ends with an arm round run on a different vendor, even though it costs more money and more time. The two lenses catch different classes of defect, and the class same-vendor misses is the expensive one to let through. A value that has touched a tracked file is burned and has to be rotated. Downstream there is nothing cheaper.

merge

AST-074promoted 2026-08-18

A tracker measured only against itself cannot detect its own drift

Reported upstream from an adapted project's own Thomas, via a handoff, rather than found in this package directly — the first entry with that provenance, recorded because the class it names is general and the reachability checks have no way to see a tracker's *content* going stale, only a document's.

Four tickets in that project sat claimed and in-progress with a live assignee **after their code had merged to the base branch**, the oldest by a full day. …

measured once, one project — 4 tickets stale, the oldest by a full day, before anyone noticed.

Four tickets sat in progress with a live assignee after their code had merged, the oldest by a full day. Nothing errored: the merge ran, the frontier write-back after it did not, and no artifact recorded the omission.

No tracker-only check catches this, because a wrong state is perfectly consistent with itself. The tracker has to be reconciled against Git after every merge instead of confirming itself. The cost is a reconcile step nobody enjoys running, and most of the time it finds nothing.

AST-056promoted 2026-08-12

A blocking edge expresses order, not exclusion

The frontier asks which tickets have no open blocker and no assignee. It never asks what each ticket will WRITE, and nothing else did either. Two tickets went out together, correct by every rule the package stated. One existed to correct WIRE-CONTRACT.md; …

measured twice in one day — a naive merge would have reverted already-reviewed work.

Two tickets had no blocking edge between them, correct by every rule I had written, and both edited the same three rows of one file. The first merged. The second was based on the commit before that merge, produced two conflict blocks, and a merge in the wrong direction would have reverted reviewed work with no signal.

One worktree per Builder solves the checkout collision and nothing else. It relocates the collision to the merge, where it is found late and by hand. Tickets now declare a write-set, and two tickets whose write-sets overlap are serialised even when nothing orders them. The cost is fewer tickets running at once, which is the exact thing this whole system exists to increase.

Explore the full ledger →