ApprenticeOS

Turn expert corrections into permanent evaluations and block deployment until every regression passes.

Comment

Inspiration

AI agents can accept a correction in one conversation and still repeat the same mistake later. Feedback is often stored as text, but it does not become an executable test that protects future releases.

ApprenticeOS explores a simple idea: treat every expert correction like a software regression. A correction should outlive the conversation, rerun against future versions, and block deployment when the behavior is still unsafe.

What it does

ApprenticeOS is an evaluation-driven repair workflow for AI agents.

The demo follows one complete loop:

  1. Teach — expert examples define expected behavior.
  2. Test — a new request exposes a policy gap.
  3. Correct — the expert supplies the safe result.
  4. Repair — the correction becomes regression test #30 and produces a minimal policy update.
  5. Verify — the full evaluation suite reruns, including the original failure.
  6. Deploy — the release gate opens only after verification succeeds, then a downloadable Agent Pack is generated.

The example uses a fictional food-aid coordinator. Policy v1 incorrectly treats “cannot eat peanuts” as a preference and skips human review. After correction and repair, policy v2 classifies it as high risk and routes the request to a human.

Why permanent evaluations matter

A saved conversation is not the same as evaluation memory.

ApprenticeOS stores the input, incorrect output, expert correction, reason, and expected behavior together as a regression asset. Every later release must pass that case again. This makes corrections measurable, reviewable, and enforceable.

How we built it

The application is built with Next.js and TypeScript. It includes:

  • a deterministic evaluation engine
  • versioned policy definitions
  • structured outputs validated with Zod
  • expert-correction and regression records
  • a minimal repair workflow
  • a verification-dependent deployment gate
  • Agent Pack ZIP generation
  • unit and integration tests
  • a server-side OpenAI Responses API integration path

The public deployment runs in Deterministic Demo Mode. It requires no API key and does not claim live model calls, allowing judges to reproduce the full workflow reliably.

How Codex and GPT-5.6 were used

GPT-5.6 Sol was used through Codex throughout development.

Codex assisted with product architecture, the Next.js implementation, the evaluation engine, policy versions, API routes, Agent Pack generation, automated tests, debugging, accessibility improvements, GitHub publishing, and Vercel deployment.

Human review remained part of the engineering process. Manual acceptance testing found that the deployment gate opened too early after repair and that some policy-state copy was inaccurate. Those findings were given back to Codex, which corrected the state machine and added regression tests covering the failure.

Human review also identified homepage contrast problems. Codex updated the styles and verified the page against WCAG AA contrast requirements.

Challenges

The main challenge was separating three states that initially looked similar:

  • the repair produced a candidate policy
  • the evaluation suite reached 30/30
  • the repaired behavior was independently verified

A 30/30 score alone must not open the release gate. ApprenticeOS keeps deployment blocked until verification confirms the original failure now produces the expected high-risk, human-review result.

Another challenge was making the public demo reproducible on a serverless deployment without requiring paid infrastructure. The deployed demo therefore uses controlled runtime state that may reset after a cold start.

Accomplishments

  • A complete Teach → Test → Correct → Repair → Verify → Deploy workflow
  • 30 evaluation cases, including a dynamically created regression
  • A deployment gate enforced in both the interface and server routes
  • A downloadable Agent Pack generated only from verified artifacts
  • 17 automated tests across 10 test files
  • Successful TypeScript, ESLint, integration, and production-build checks
  • 11 routes generated successfully
  • A public Vercel deployment and an open-source GitHub repository

What we learned

Corrections become valuable when they are converted into executable evidence.

Reliable agents need more than better prompts. They need evaluation memory, versioned behavior, verification, human oversight, and release gates similar to the safeguards used in traditional software engineering.

What's next

Future work could add:

  • persistent team workspaces and databases
  • signed evaluation provenance and reviewer identities
  • domain-specific evaluation packs
  • model-generated repair candidates reviewed by humans
  • richer adversarial and multilingual evaluations
  • audit histories across multiple agent releases

Safety and limitations

All people, requests, and organizations shown in the demo are fictional.

The public demo is not a production food-aid or medical decision system. It runs deterministically, does not make live OpenAI API calls, and uses in-memory runtime state that may reset. In a production system, repairs would require authorized review, durable storage, monitoring, and domain-specific safety validation.

Built With

Share this project:

Updates