Context MRI

Find the context file that broke your agent. Verify the repair. Block its return.

Comment

Context MRI

Most evaluations tell you that an agent failed. Context MRI tests which supplied context changed the result, helps Codex stage the smallest evidence-backed repair, verifies the repaired pack, and creates a CI guard so the regression cannot return.

Try the free no-login demo · View the source and installation guide · Inspect the public CI proof

Inspiration

AI agents do not fail only because they lack context. They also fail because they have too much: an obsolete README, duplicated rules, irrelevant examples, or tool instructions that quietly contradict current policy. Developers usually debug this by rereading prompts and guessing.

Context MRI turns that guesswork into a controlled experiment.

What it does

Context MRI takes a task, a task-specific evaluator, and a bundle of context files. It runs the full bundle as a baseline, then removes one file at a time while keeping the task and rubric fixed. Every condition is repeated. The result is an ablation matrix showing which context helps, appears neutral, or has the largest observed negative single-file effect for this task under this evaluator.

The workflow continues past diagnosis:

  1. Diagnose — compare the full bundle with one-file-removed conditions.
  2. Inspect — open exact run IDs, prompt hashes, model outputs, token use, latency, and rubric-level scores.
  3. Repair — stage the smallest evidence-backed context change.
  4. Verify — rerun the reduced pack against the same task and evaluator.
  5. Prevent — export a fingerprinted Context Guard for CI.

The featured Security Release diagnostic includes an unsafe legacy runbook that conflicts with current credential policy. Its bundled replay moves from a 53-point baseline to a verified 100-point repaired pack when that runbook is removed. Context Guard then proves both sides of the gate: the original pack remains blocked and the repaired pack passes.

The public demo also includes Support and Billing diagnostics with independent contracts and datasets. For genuinely new work, Local Judge Lab lets a developer define a task, exact success criteria, conflicting instruction, and source labels for the files loaded in the browser. It requires a fresh local API run; there is no custom-fixture fallback that makes an arbitrary result look proven.

Native Codex workflow

Context MRI also ships as a free local Codex plugin. A developer can ask Codex to diagnose a supplied pack, inspect the evidence, propose a repair through the normal approval flow, and verify the before/after result without leaving the coding conversation.

The plugin exposes three narrowly scoped, read-only MCP tools over local stdio. It makes no external network requests, retains no files, and never edits a repository itself. Those are plugin boundaries; Codex and model execution still follow the user's configured OpenAI services.

Tracing tools show what an agent did. Context MRI experimentally tests which supplied context changes the result, then verifies the repair.

How we built it

The product combines:

  • React, TypeScript, and Vite for the judge-facing application
  • A Node/Express experiment service
  • A Cloudflare-compatible public fixture adapter
  • A transport-neutral diagnostic core shared by the web app and Codex plugin
  • Three read-only MCP tools over local stdio
  • An optional OpenAI Responses API runner using GPT-5.6 Sol and strict Structured Outputs
  • A deterministic, task-specific evaluator that assigns every rubric point independently of the subject model
  • A portable Context Guard and zero-secret GitHub Actions workflow

Every trace records its evaluation contract, run ID, condition, repeat, prompt hash, provenance, token use, latency, model output, and rubric breakdown. Public replay metrics are derived from those trace records rather than typed into the interface.

The installed plugin returns a short-lived guard reference for reliable same-session verification, while the exported guard fingerprints the contract, source pack, and evidence artifact for portable CI use. The check fails on a blocked term, a score below threshold, or an integrity mismatch.

Built with Codex and GPT-5.6

Codex helped take Context MRI from a critical product question to a working, tested developer tool: researching the official requirements, comparing product directions, shaping the architecture, implementing the interface and MCP service, writing automated tests, checking the math behind every displayed claim, running browser QA, packaging the plugin, and hardening the public proof.

GPT-5.6 Terra in Codex was used for the final adversarial review of the evaluator, fixture claims, privacy boundary, judge flow, and submission evidence. That review removed a false paid-API judging gate and tightened the distinction between observed evidence and general causation. The optional fresh runner uses GPT-5.6 Sol through the Responses API.

Evidence and credibility boundaries

The hosted judge path is free and deterministic: no account, payment, API key, or external setup. It is clearly labeled as fixture replay evidence, not a fresh live-model claim. Fresh API evaluation is an optional local path, and unavailable funded controls fail explicitly instead of silently substituting replay data.

Each bundled public replay contains 21 inspectable traces: 18 discovery traces plus three independent repaired-pack verification traces. Thirty-six automated tests protect the evaluator, aggregate claims, provenance, guard behavior, plugin service, sanitized proof contract, and real MCP transport.

Five fresh Codex tasks separately tested plugin discovery and orchestration. All five completed one diagnosis and two-sided guard verification. The public record contains only allowlisted metadata: tool names, outcomes, timestamps, and hashes.

We also publish the robustness boundary:

  • A lexical robustness check changes the file ID, filename, and surrounding prose while preserving the evaluator's configured blocked phrase; Context MRI still finds the 53→100 effect.
  • A semantic-paraphrase negative control removes that phrase and is not detected by the deterministic fixture.

That miss is public because the correct claim is controlled, task-specific ablation evidence, not universal causation or broad semantic generalization.

Challenges

The hardest challenge was epistemic honesty. A polished dashboard can make weak evidence look stronger than it is. We removed an early confidence percentage that three repeats could not justify, corrected inconsistent claims, exposed replay/live provenance in the interface, and made unsupported custom tasks fail honestly.

Dogfooding the release process also found two genuine inconsistencies: an upload checklist referenced an older video, and the original CI only proved that the repair passed. The final workflow now asserts both outcomes, fingerprints the audited files, validates the plugin over real MCP stdio, and publishes inspectable JSON artifacts.

Accomplishments

  • A complete diagnosis → repair → verification → prevention loop
  • A free, local-first Codex plugin with three read-only tools
  • Original pack blocked and repaired pack passed in public CI
  • Three separately configured diagnostic scenarios
  • Independent evaluator scoring rather than model self-grading
  • Five of five fresh Codex orchestration runs completed
  • A published lexical robustness check and semantic negative control
  • Thirty-six passing automated tests
  • A free, no-login public judge path

What we learned

Context quality is not context quantity. The most damaging file can look authoritative, and token reduction alone is not proof that removal is safe. A useful context debugger must connect every recommendation to a stated task, an inspectable evaluator, and a repeatable verification step.

What's next

Next are broader multi-file interaction tests, production trace imports, human-calibrated rubrics, and adapters for common agent frameworks. The plugin will remain free. A future hosted layer could add team history, private deployment, and fleet-level CI only after strong retention, deletion, and tenant-isolation controls are in place.

The long-term vision is a context compiler: route only the minimum context proven useful for each task, and continuously guard that decision as agents and repositories change.

Built With

Share this project:

Updates