A risk assessment of nine months of daily AI-assisted building — 392 annotated failure episodes across three machines and twelve projects, scored the way you'd score any risk register: frequency × severity.
Same shape as any third-party risk assessment: inventory, categorise, score, find the tail.
Every Claude Code session across three machines was archived — 5,842 session logs. Moments
where I pushed back, corrected, or rejected the work were extracted as episodes, then
annotated with a failure category, a missing discipline, a severity (1–3), and the verbatim
quote that triggered it. Score is Σ severity, so a category earns its rank
through both how often it happens and how much it costs.
The categories are deliberately about practice, not about tooling. "The model was wrong" is not a category. "I never said what I wanted" is.
392 episodes, nine categories, ranked by score.
| Category | Episodes | Avg severity | Score | Weight |
|---|---|---|---|---|
| requirements_intent | 160 | 2.08 | 332 | |
| architecture_design | 121 | 2.07 | 250 | |
| spec_clarity | 33 | 2.09 | 69 | |
| observability_ops | 27 | 2.04 | 55 | |
| testing_verification | 18 | 2.22 | 40 | |
| scope_control | 13 | 2.38 | 31 | |
| change_management | 9 | 2.11 | 19 | |
| security_secrets | 5 | 3.00 | 15 | |
| debugging_method | 6 | 2.00 | 12 |
requirements_intent (160) + architecture_design (121) + spec_clarity (33) = 314 of 392 episodes — 80%. Four out of five times something went wrong, the cause was that I had not decided, or had not said, what I actually wanted. The building was rarely the problem.
"can we stop doubling up and make the product to customer fit / request more prominent"
— requirements_intent
This is the finding that changed how I work. Speed of generation makes under-specification cheaper to commit and more expensive to discover.
Across 199 delegation observations: 130 normal, 56 effective delegation, 5 micromanagement, 4 self-performed, and 4 where the agent outright failed — about 2%. The tool was not the bottleneck. The instruction was.
security_secrets: 5 episodes — the rarest category — at average severity 3.00, the highest by a wide margin (everything else sits between 2.00 and 2.38). Textbook low-frequency, high-impact. You cannot drive it down with practice; you drive it down with a control.
"any way we can get past captcha? why don't you launch the page, give me a 20 second head start"
— security_secrets, severity 3
And a detail I enjoy more than I should: one evidence quote in the raw dataset contains a live OAuth client identifier, pasted into a chat window. The report documenting my secrets-hygiene problem would itself have leaked a credential. It is redacted here. That is the category behaving exactly as described.
A risk report that doesn't state its limits is marketing. These are the ones that matter:
A finding that doesn't produce a control is just a complaint.
Specify before generating. Acceptance criteria written down before the work starts — because 80% of the register says that is where the cost is.
Encode the criteria as a test, not a memory. Design contracts now ship with automated rubrics that grade generated output against them and fail the build on regression — and I validate the rubric by seeding deliberate defects, so I know it detects rather than rubber-stamps.
Treat the tail risk with a control, not with care. Secrets never transit a chat window; they come from a vault or an environment.