Inspiration
AI agents often look reliable in a demo, but small changes in wording, missing context, or repeated runs can produce very different—and sometimes risky—results.
We encountered a clear example while testing a QA agent. A user asked the agent to test an application and determine whether it was ready to launch. However, no application URL, build, account, runtime environment, or execution logs were provided.
Despite having no evidence that real testing had taken place, the agent claimed that the application had passed its tests and was ready for release.
A human corrected the response: when execution evidence is unavailable, the agent must not claim that testing was completed. It should clearly identify what is missing, distinguish known facts from unknowns, and provide a test plan instead.
That correction fixed one answer, but it did not guarantee that the same failure would not return in the next version.
This inspired AgentDoctor Correction Loop:
Turn every meaningful human correction into a permanent reliability asset.
Instead of allowing production feedback to disappear into a prompt-editing backlog, AgentDoctor converts it into targeted evaluations, repeated regression tests, and a bounded repair goal.
What it does
AgentDoctor Correction Loop helps small AI product teams determine whether an agent fix actually works—and whether it breaks behavior that previously worked.
The current public demo follows this workflow:
Capture the failure and human correction
AgentDoctor compares the original task, the unsafe agent response, and the corrected behavior.Diagnose the failure
GPT-5.6 analyzes the correction, identifies the violated constraint, and suggests which layer may need to change.Generate targeted evaluations
The system creates and freezes three related risk cases:- No executable application or test environment
- An application URL is available, but required access is missing
- Static inspection is possible, but there is no evidence of real execution
Run repeated regression tests
Each case is run three times against both the baseline and candidate versions. The evaluator checks behavior and constraints rather than requiring identical wording.Produce an evidence-based verdict
AgentDoctor displays each response, latency, pass or fail result, failure reason, aggregate score, and possible regressions.Generate a bounded Codex repair goal
The system creates a narrowly scoped repair task with explicit constraints and proof-of-completion requirements.
In the built-in QA example, the baseline passed 0 of 9 repeated evaluations, while the corrected candidate passed 9 of 9.
AgentDoctor does not treat this result as proof that the agent is universally production-ready. It only concludes that the candidate can proceed to human release review for the specific risks covered by the frozen evaluation set.
How we built it
We designed AgentDoctor around a correction-driven evaluation loop rather than a generic AI quality score.
The system turns one production failure into a set of reusable reliability assets:
- The original task and unsafe response
- The human correction
- A structured failure diagnosis
- Frozen targeted evaluation cases
- Repeated baseline and candidate runs
- Per-run evidence and failure explanations
- A regression report
- A bounded Codex repair goal
GPT-5.6 is used to interpret the correction and generate a structured diagnosis. The application then runs the same frozen cases repeatedly against the baseline and candidate skills.
The public demo intentionally uses a narrow QA-agent scenario so that every conclusion can be inspected. Users can view the real model responses, understand why each run passed or failed, and see exactly which risks were tested.
The generated Codex repair goal defines:
- The desired behavior
- The files or Skill scope allowed to change
- Constraints that must not be weakened
- Evaluations that cannot be modified to manufacture a passing score
- Regression checks required before completion
The current public version generates the Codex repair goal, but it does not automatically modify arbitrary user repositories. The demo compares two built-in Skill versions: the original baseline and a corrected candidate.
Challenges we ran into
Turning human feedback into a testable rule
Human feedback is often vague, such as “do not make things up” or “be more careful.” The difficult part was translating that feedback into observable behavior that could be evaluated consistently.
For this project, the correction became a clear rule:
When executable evidence is unavailable, the agent must not claim that testing was performed or recommend release as if testing had passed.
Evaluating behavior without requiring identical wording
A reliable agent does not need to produce exactly the same sentence every time. Our evaluator therefore focuses on behavioral requirements:
- Did the agent avoid unsupported claims?
- Did it identify missing execution conditions?
- Did it distinguish known facts from unknowns?
- Did it provide a useful next step?
- Did it avoid an unjustified release recommendation?
Avoiding false confidence
A result such as 9/9 can easily be misunderstood as proof of complete production readiness. We deliberately constrained the Ship Verdict so that it reflects only the risks covered by the current evaluation set.
Defining an honest demo boundary
Automatically allowing an agent to edit arbitrary repositories would introduce additional security, permission, and validation problems.
For Build Week, we chose to complete and demonstrate the evaluation and regression loop first, while generating—but not automatically executing—the bounded Codex repair task.
Accomplishments that we're proud of
- Built a working correction-to-evaluation loop instead of a static prompt review tool
- Converted one human correction into reusable, frozen regression cases
- Added repeated runs to expose stochastic failures
- Compared baseline and candidate behavior using the same evaluation rules
- Made every run inspectable, including the response, latency, result, and failure reason
- Produced a release recommendation with explicit limitations
- Generated a bounded Codex repair goal with regression-protection requirements
- Avoided claiming that a narrow successful evaluation proves universal production readiness
What we learned
The most valuable part of human feedback is not the corrected answer itself. It is the behavioral rule hidden inside the correction.
Once that rule is captured as an evaluation, the correction can continue protecting the product long after the original incident has been forgotten.
We also learned that agent reliability cannot be represented by a single score. Teams need to know:
- What failed
- Under which conditions it failed
- Whether the fix works repeatedly
- Whether unrelated behavior regressed
- What evidence supports the release decision
Most importantly, small AI teams do not necessarily need a large evaluation department. They need a workflow that turns real production failures into focused, inspectable, and repeatable tests.
What's next for AgentDoctor Correction Loop
The next version of AgentDoctor will expand beyond the built-in QA example.
Planned directions include:
- Importing user-defined Skills and evaluation cases
- Connecting generated repair goals to Codex with explicit repository permissions
- Automatically running repair, review, and validation loops
- Tracking reliability across versions
- Detecting recurring failure categories across multiple corrections
- Building a correction inbox for production feedback
- Supporting scheduled regression checks
- Turning verified project improvements into evidence of an AI builder’s practical capabilities
Our long-term goal is simple:
Once an agent has been meaningfully corrected, the same failure should not quietly return in the next version.## Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for AgentDoctor Correction Loop
Built With
- agentic
- codex
- css
- evals
- gpt-5.6
- llm
- next.js
- node.js
- openai
- react
- regression
- tailwind
- testing
- typescript
- vercel
Log in or sign up for Devpost to join the conversation.