Amadeus Skill

Amadeus is a local-first control plane for coding agents. It compiles human-approved runs into compact skills via Codex CLI (no API key needed) for secure, opt-in procedural memory.

Comment

Amadeus Project Story

Amadeus turns verified Codex runs into agent skills that earn their place.

Inspiration

Coding agents solve useful problems every day, but most of that procedural knowledge disappears when the session ends. Saving every successful answer as a permanent instruction is not enough: a procedure can be correct once and still be too narrow, too expensive, unsafe, or likely to trigger on the wrong task.

We built Amadeus to answer a harder question:

What did the agent learn that is reusable, and did that learning measurably earn a permanent place?

What Amadeus does

Amadeus is a local-first control plane for procedural memory. It captures an explicitly consented Codex run, distills the verified resolution into a compact Agent Skill or deterministic script, and keeps that artifact as a candidate until humans and evidence approve it.

The lifecycle is:

  1. Capture an opt-in, session-scoped Codex run.
  2. Distill verified work into a skill, script, or mixed artifact.
  3. Review the generated package with a named human and rationale.
  4. Validate triggers, anti-triggers, termination rules, safety, structure, and compactness.
  5. Evaluate matched skill-disabled and skill-enabled trials plus negative cases that must abstain.
  6. Promote only when capability, safety, reliability, and economic gates pass and a human gives final approval.
  7. Retrieve active skills using compatibility filters, FTS5, deterministic similarity, a confidence threshold, a margin, and an abstain path.
  8. Observe or retire procedures using stored outcomes, provenance, rollback, and an audit trail.

Candidate packages, evaluation evidence, the retrieval index, metrics, and exported Agent Skills remain separate. A model-generated candidate never activates itself.

What makes it different

Amadeus is not another prompt library and it is not long-term chat memory.

  • Outcome-first: it grades whether the task was verified, not whether the agent followed one exact trajectory.
  • Reliability-first: every treatment capability case must pass, negative cases must abstain, protected paths must remain intact, and unsafe attempts fail the gate.
  • Value-aware: a procedure must improve success or retain perfect success with material efficiency value, without regressing a measured economic metric by 20% or more.
  • Human-governed: candidate creation, testing approval, and final promotion are separate explicit decisions.
  • Local-first: source, SQLite data, candidate packages, evaluation results, provenance, and metrics stay in the project.
  • Progressive disclosure: metadata is considered first; the full skill body and focused resources load only when selected.
  • Harness-ready: Codex is the first adapter, while redaction, governance, candidate, registry, retrieval, and evaluation contracts are reusable by future agent harnesses.

How we built it

The hackathon implementation is a single npm-distributable TypeScript product:

  • native Codex CLI using the user's existing ChatGPT authentication;
  • GPT-5.6 through the locally advertised gpt-5.6-sol coding model;
  • a harness adapter that captures redacted observable events without storing model reasoning;
  • Fastify for the local API and dashboard server;
  • React for the procedural-memory dashboard;
  • Node's native SQLite plus FTS5 for durable local state and retrieval;
  • safe regular-expression filters and deterministic hashed similarity;
  • an MCP stdio server connecting normal Codex sessions to Amadeus evidence;
  • Agent Skills packages with SKILL.md and agents/openai.yaml;
  • an npm tarball workflow for installation in another project before registry publication.

Amadeus never asks for or stores an OpenAI API key. Every Codex child process strips OPENAI_API_KEY and CODEX_API_KEY and relies on the existing local ChatGPT login.

The real tests changed the product

We did not stop at a demo fixture. We installed the packaged CLI into a cloned React project and used it on real failures.

Legacy Node Sass on Apple Silicon

Codex migrated an old React build from unsupported Node Sass tooling to Dart Sass. Baseline and treatment both built successfully, and the exact frozen candidate load was proven. Treatment was faster, but input-token usage regressed beyond the economic guardrail. The skill remained blocked.

Stale asynchronous UI results

We introduced deterministic tests for two common races: requests completing out of order and a pending request completing after the search was cleared. Baseline and treatment both fixed the behavior, and a synchronous-filter negative case correctly abstained. The treatment nevertheless used more input tokens, more time, and more tool calls. Amadeus rejected promotion.

Those are not failed demos. They demonstrate the central product promise: successful invocation is not sufficient evidence of durable value.

The repository preserves all three governed candidates—SQLAlchemy detached instances, legacy Node Sass migration, and stale async UI results—with honest registry, review, and evaluation status. None is falsely presented as active.

Challenges

Proving that a skill was actually used

Placing a file near Codex did not prove causality. We added frozen treatment snapshots plus an MCP receipt that verifies candidate ID, content hash, and prepared path before a treatment trial can be recorded.

Separating testing from activation

Early testing exposed candidates through .agents/skills, which blurred controlled availability and production activation. We changed the design: testing uses isolated .amadeus/evaluations snapshots, while .agents/skills is written only after measured gates and final human approval.

Keeping non-interactive runs safe and observable

A treatment run stalled on an MCP approval that no terminal could answer. We now approve only the intentional skill-load receipt tool in evaluation repositories and apply a bounded Codex process-group timeout. Other sandbox, network, and tool boundaries still fail closed.

Measuring value fairly

End-to-end treatment cost includes discovery, skill instructions, and the governance receipt. That is honest product overhead, but one matched pair is too noisy to isolate solve-only value. Amadeus reports the measured result without turning it into a stronger causal claim.

What we are proud of

  • A complete local capture → candidate → review → validation → controlled evaluation → promotion-gate vertical slice.
  • Real package installation and use in another repository.
  • Verifiable skill-load provenance rather than relying on a prompt claim.
  • Negative-trigger and abstention support.
  • A strict separation between candidates and active exported skills.
  • Honest evaluation results that blocked every candidate that had not earned promotion.
  • A focused core suite that verifies redaction, SQLite/FTS5, safe regex, candidate isolation, and post-gate activation.

Current status

The v1.1 implementation is machine-verified and has been manually exercised from package installation through controlled evaluation. The npm tarball works for private distribution; registry publication is still pending. Codex is the only live harness today. No current candidate has passed every promotion gate, so the active skill count correctly remains zero.

What is next

  • Publish the CLI as an npm package.
  • Add an audited candidate-revision workflow after human feedback.
  • Separate end-to-end governance overhead from solve-only evaluation metrics.
  • Run repeated matched cases across unseen repositories to reduce variance.
  • Add more harness adapters while preserving the same governance contracts.
  • Promote the first skill only when the measured evidence and a named human both say it earned its place.

Built with Codex

Codex accelerated the TypeScript implementation, native SQLite migration, Fastify/React integration, packaging, skill generation, real-project fixes, tests, documentation, and evidence capture. Humans made the key product decisions: opt-in capture, no API keys, candidates never self-activate, testing stays isolated from active skills, economic regressions block promotion, and failed-value skills remain visible rather than being hidden.

Built With

  • agent-skills
  • ai-coding-agent
  • apple-silicon
  • code-review
  • codex
  • codex-cli
  • dashboard
  • developer-tools
  • fastify
  • fts5
  • gpt-5.6
  • local-first
  • macos
  • mcp
  • node.js
  • npm-package
  • openai
  • procedural-memory
  • react
  • skill-evaluation
  • software-architecture
  • software-automation
  • sqlite
  • typescript
  • zod
Share this project:

Updates