Prompt Autopsy

Investigate failed AI coding sessions, trace findings to evidence, compare historical agent rules, and decide what should change—without letting the model make the final call.

Comment

Inspiration

AI coding agents can produce fixes that look correct, pass automated checks, and still fail during real use. When that happens, it is easy to respond by adding another rule to AGENTS.md or CLAUDE.md, even when the evidence does not actually show that missing instructions caused the failure.

I built Prompt Autopsy to make that process more deliberate.

Instead of immediately changing an agent's instructions, Prompt Autopsy investigates one failed coding session, separates evidence from assumptions, compares the incident with the historical instruction file, and lets the developer decide whether a permanent rule change is justified.

What it does

Prompt Autopsy takes three inputs:

  • a description of the coding incident;
  • a structured transcript of the developer and coding-agent session;
  • the historical AGENTS.md or CLAUDE.md from that incident.

Before any AI analysis happens, the developer reviews and can redact the exact information that will be processed.

After consent, Prompt Autopsy uses OpenAI to interpret the incident and creates an evidence-backed investigation containing:

  • an annotated timeline;
  • documented requirements;
  • observations linked to exact transcript evidence;
  • possible explanations kept separate from established facts;
  • missing evidence;
  • comparison with historical instructions;
  • an add, edit, no-change, or needs-evidence recommendation.

If an instruction change is justified, Prompt Autopsy proposes the change, but it does not automatically modify the developer's repository.

The developer can approve, reject, accept no change, or request more evidence. Only explicitly approved changes are included in the exported AGENTS.md or CLAUDE.md.

It also exports a Markdown investigation report containing the evidence, findings, decisions, limitations, and proposal history.

How I built it

Prompt Autopsy is built with Next.js, React, TypeScript, Zod, and the OpenAI Responses API.

I deliberately separated model reasoning from deterministic application validation.

OpenAI handles interpretation tasks such as understanding the incident, grouping evidence into findings, comparing the failure with historical instructions, and proposing narrowly scoped rule changes.

Application code then verifies things that should not depend on model judgment, including:

  • exact transcript references;
  • exact quoted evidence;
  • quote occurrences;
  • historical rule references;
  • proposal relationships;
  • edit anchors and boundaries;
  • replacement size limits;
  • structured-output consistency.

If the model returns structurally valid output that fails those checks, Prompt Autopsy rejects the investigation rather than silently accepting or repairing it.

The core design principle became:

AI interprets. Application code verifies. Developer decides.

Challenges

The hardest part was discovering that structured output is not the same as verified output.

During real provider testing, OpenAI could return valid structured JSON while the application later rejected the result because a citation, proposal relationship, or edit target did not satisfy the reviewed source material.

Instead of weakening those checks, I progressively moved provider-expressible invariants into the Structured Output schema and introduced typed, safe deterministic validation errors for the checks that must remain application-side.

Another challenge came from the product itself. A browser-restoration fix in Prompt Autopsy passed offline tests, typecheck, and a production build but introduced a real browser regression. That incident became one of the real cases used to test Prompt Autopsy.

Accomplishments

I am especially proud that Prompt Autopsy does not force every incident into a rules change.

A successful investigation can conclude:

  • add a rule;
  • edit an existing rule;
  • make no change;
  • or gather more evidence.

The real browser-regression investigation correctly found that the historical AGENTS.md did not contain relevant React-controlled-input guidance, proposed a new instruction outside the generated Next.js block, and required explicit developer approval before export.

The project currently passes 206 offline tests, TypeScript validation, and a production webpack build.

What I learned

The biggest lesson was that valid AI output still needs deterministic verification.

A model can return valid JSON and still reference evidence incorrectly or generate an edit that does not match the reviewed source.

I also learned that automated tests are not proof of real-world behavior. One of the actual failures used to test Prompt Autopsy passed offline tests, typecheck, and production build before hands-on browser testing exposed the regression.

That changed how I think about AI developer tools. Uncertainty should not be hidden. Sometimes the most accurate result is not a confident fix, but a clear statement that more evidence is needed.

What's next

The current version focuses on one manually prepared coding incident at a time.

Future work could include safer integrations for importing coding-session history, repository context, commit diffs, and browser/test evidence while preserving the same evidence-review and developer-approval model.

Built With

Share this project:

Updates

Submission history