FailSpec
Inspiration
Over the last few years, coding with LLMs has gone from feeling almost impossible to feeling completely normal.
As students, we have used AI to understand unfamiliar codebases, generate components, debug errors, and build things much faster than we could before. It is honestly exciting. A motivated student can now sit down with an idea, open an editor, talk to an agent, and build something surprisingly real.
And students are getting extremely good at it.
But we also kept running into the less exciting side of AI-assisted coding.
You tell the model that something is broken. It suggests a fix. The fix causes another issue. You paste the new error. It changes three more files. Eventually, you are ten prompts deep, the original bug still exists, and neither you nor the model are completely sure what actually went wrong.
That is the trap of endless AI debugging loops.
The problem is not always that the model cannot write the fix. The problem is that the failure was never properly defined or reproduced in the first place.
A bug report such as “checkout is broken” or “the total is wrong” gives an agent a lot of room to guess. It may produce a convincing explanation and a plausible patch, but that does not mean it understood the real issue.
We wanted to move beyond vibe coding.
As AI writes more of our code, we need better ways to verify what it is doing. Before asking an agent to fix a bug, we should be able to answer:
What exactly is broken, can we reproduce it, and what evidence would prove that it has been fixed?
That is why we built FailSpec.
FailSpec turns vague software failures into executable, evidence-backed regression tests. It gives developers and coding agents a shared, testable definition of the problem before anyone starts changing code.
What it does
A developer gives FailSpec:
- A trusted local repository
- A description of the failure
- The expected behavior
- The observed behavior
- Any useful technical or reproduction context
FailSpec then performs a structured investigation:
- It validates and inspects the repository.
- It analyzes the report and relevant parts of the codebase.
- It creates a concrete reproduction hypothesis.
- It generates a focused Playwright regression test.
- It runs the test in an isolated environment.
- It collects structured execution evidence.
- It classifies the outcome as:
verifiedpartialnot_reproducedexecution_error
The output is not just an AI response saying, “I found the problem.”
It is a reviewable hypothesis, a generated test, an execution result, and a verdict grounded in what actually happened.
The resulting regression test can then become the target for either a human developer or another coding agent. Instead of giving an agent a vague instruction such as “fix checkout,” we can give it a failing test and a clear condition for success.
How we built it
FailSpec is a local-first application built around a clear separation between AI reasoning and actual verification.
The frontend uses React, TypeScript, and Vite. It allows developers to submit a failure report, follow the investigation through a visible lifecycle, and inspect the final verdict and its supporting evidence.
The backend uses Node.js and Express as the local orchestration layer. It manages:
- Repository preflight and trust checks
- Investigation lifecycle state
- Codex-powered repository analysis
- Reproduction-hypothesis generation
- Playwright test generation
- Controlled test execution
- Evidence normalization
- Local investigation persistence
- Deterministic verdict classification
Each investigation runs inside an isolated Git worktree.
This was important to us because an AI investigation should not be freely editing or staging files inside the developer’s active working copy. The worktree gives FailSpec a separate environment where it can generate and run the reproduction test while leaving the original repository untouched.
Investigation records are stored locally as structured JSON, allowing the workflow to remain inspectable without requiring developers to upload private repositories to an external platform.
One of our main architectural decisions was that the model should not be allowed to simply grade its own output.
The model is responsible for reasoning about the failure and proposing a reproduction. The test runner is responsible for executing it. The evidence layer records what happened. A deterministic, side-effect-free classifier then produces the final verdict from typed and sanitized execution evidence.
In other words, the AI can propose the experiment, but the actual execution determines what we can claim.
Challenges we faced
Deciding what counts as real verification
At first, it seemed like a failing test should mean the bug had been reproduced.
But we quickly realised that this was not reliable enough.
A test can fail because:
- The application never started
- A selector was wrong
- The test timed out
- The generated test misunderstood the report
- The expected and actual values differed for an unrelated reason
- The runner itself failed
We had to distinguish between a genuine reproduction, partial supporting evidence, a valid test that did not reproduce the bug, and an invalid execution.
Designing the verdict system and its precedence rules became one of the hardest and most important parts of the project.
Avoiding another opaque AI tool
Many AI tools follow the same pattern: enter a prompt, wait, and receive a confident answer.
We did not want FailSpec to become another black box.
The investigation timeline exposes the main stages of the process, and the result screen connects the verdict to the generated hypothesis, test, exit code, execution status, and observed evidence.
The developer should be able to disagree with the system and understand why it reached its conclusion.
Converting flexible model output into strict software contracts
LLMs are flexible and conversational. Production systems are not.
Our orchestration pipeline needed structured data that could safely pass between analysis, hypothesis generation, test creation, execution, evidence collection, and classification.
We had to build typed boundaries, validate model outputs, sanitize evidence, and keep unreliable text away from deterministic logic.
Keeping repository execution safe
FailSpec has to inspect and execute a real application, which introduces real risks.
The isolated worktree reduced the chance of modifying a developer’s active workspace, but it required careful handling of paths, generated files, process execution, cleanup, and repository state.
What we learned
The biggest thing we learned is that generation and verification are completely different problems.
It is easy for an LLM to generate something that looks correct. It is much harder to prove that the result corresponds to the real bug, ran under valid conditions, and produced evidence strong enough to trust.
We also learned that the best way to make an AI system more trustworthy is not simply to improve the prompt. It is to improve the system around the model.
FailSpec separates the workflow into different responsibilities:
- The model proposes how the bug could be reproduced.
- The runner executes the generated test.
- The evidence layer records what actually happened.
- The deterministic classifier decides what the evidence supports.
- The developer can inspect every stage.
This separation prevents the system from producing both the answer and the justification for that answer.
More broadly, the project changed how we think about AI-assisted development.
Vibe coding is powerful for exploration and fast prototyping. But when a project becomes real, developers need to move from “this seems to work” to “we can prove this works.”
AI has made it easier than ever to produce code. The next challenge is building the trust infrastructure around that code.
What’s next
FailSpec currently focuses on turning reported failures into Playwright regression tests and evidence-backed verdicts.
Next, we want to add:
- Support for more testing frameworks and repository types
- Screenshots, browser traces, logs, and richer execution evidence
- Direct handoff of verified tests to coding agents
- Automatic re-execution after an agent proposes a fix
- Before-and-after evidence comparison
- CI and pull-request integration
- Shared investigation histories for development teams
Our long-term goal is for FailSpec to become a trust layer for agentic software development.
We do not want AI coding to become an endless loop of prompts, guesses, and increasingly confusing patches.
We want developers to be able to say:
This is the failure. This test proves it. Now fix it without breaking anything else.
AI can generate the change. FailSpec proves the problem and defines what success looks like.
Built With
- codex
- css
- developer-tools
- express.js
- git
- html
- json
- node.js
- npm
- openai
- playwright
- react
- rest-api
- typescript
- vite
- vitest
Log in or sign up for Devpost to join the conversation.