Codex Benchmark Guardian

A developer tool that detects performance regressions, creates bounded Codex repair goals, and verifies fixes through protected checks before merge.

Comment

Codex Benchmark Guardian

A developer tool that detects performance regressions, creates bounded Codex repair goals, and verifies fixes through protected checks before merge.

Inspiration

Functional tests can pass while performance quietly gets worse.

A pull request may preserve correct behavior but increase latency, consume more memory, slow down an important workflow, or reduce throughput. These regressions are often discovered only after deployment, when they are more expensive and risky to investigate.

Many benchmark tools can compare numbers or fail a CI job, but the workflow often stops at the alert. Developers must still determine why the change matters, investigate the likely cause, prepare a focused repair task, verify that the repair genuinely restored performance, and ensure that the benchmark policy was not weakened merely to obtain a passing result.

I built Codex Benchmark Guardian to close that gap.

It turns benchmark evidence into a clear release decision, actionable triage, a bounded Codex repair goal, and protected verification before merge.

The project follows one central principle:

Codex may help investigate and repair a regression, but it must not control the benchmark policy, manipulate the evidence, approve its own work, or merge the pull request.

What it does

Codex Benchmark Guardian compares baseline and current benchmark evidence and determines whether a proposed change introduces a material performance regression.

It supports metrics where higher values are worse, including:

  • latency;
  • runtime;
  • memory usage;
  • error rate.

It also supports metrics where lower values are worse, including:

  • throughput;
  • accuracy;
  • recall;
  • success rate.

For every comparison, Guardian can produce:

  • percentage changes;
  • regression decisions;
  • severity classifications;
  • deterministic release-readiness scoring;
  • metric-specific triage guidance;
  • Markdown and HTML reports;
  • a GitHub issue handoff;
  • a bounded Codex Goal;
  • an immutable Repair Contract;
  • CI and pull-request guardrail artifacts.

The complete workflow is:

benchmark evidence → regression decision → triage → bounded Codex repair → protected verification → human approval

The project can be used through:

  • a production Next.js dashboard;
  • a FastAPI analysis endpoint;
  • a Python command-line interface;
  • an additional Streamlit interface;
  • a protected GitHub pull-request benchmark gate.

Production experience

The primary product experience is a production dashboard built with Next.js 15, TypeScript, Tailwind CSS, FastAPI, and a deterministic Python analysis engine.

The results are organized into five sections:

  1. Overview
  2. Metrics
  3. Triage
  4. Codex Goal
  5. Handoff Pack

Developers can:

  • choose from permanent example scenarios;
  • upload or edit baseline and current benchmark JSON;
  • configure per-metric directions;
  • choose a regression threshold;
  • inspect readiness, severity, and metric changes;
  • review investigation guidance;
  • copy or download the generated Codex Goal;
  • inspect the complete Handoff Pack;
  • replay a real pull request from regression to verified repair.

The frontend does not duplicate the benchmark or repair policy. It sends the evidence to FastAPI, while the Python engine remains the only source of truth.

The verified Codex Repair Loop

The central feature of the project is the verified Codex Repair Loop.

When a material regression is detected, Guardian builds an immutable Repair Contract from the benchmark evidence.

The contract defines:

  • whether a repair is required;
  • the exact evidence requiring attention;
  • the repair objective;
  • actions Codex is allowed to perform;
  • actions Codex is forbidden from performing;
  • required validation commands;
  • protected completion criteria;
  • conditions under which Codex must stop and report.

For a genuine regression, Codex may:

  • inspect the relevant implementation and history;
  • form evidence-backed root-cause hypotheses;
  • implement the smallest maintainable correction;
  • add or update regression tests;
  • run approved project checks;
  • rerun the relevant protected benchmark;
  • review the final diff.

Codex may not:

  • lower or bypass regression thresholds;
  • change metric directions to hide a failure;
  • modify or replace the original benchmark evidence;
  • selectively choose a passing run;
  • weaken the protected harness or evaluator;
  • delete or relax tests merely to pass;
  • hard-code expected benchmark results;
  • declare the pull request Ready;
  • merge automatically.

A repair is complete only when:

  • all required project checks pass;
  • fresh evidence is produced by the protected workflow or trusted evaluator;
  • the fresh evidence reports zero material regressions;
  • the release status is Ready;
  • the final diff has been reviewed;
  • a human approves the merge.

This prevents the repair process from changing the rules used to judge its own success.

Verification-only behavior

A trustworthy agent workflow must also know when not to change code.

When valid benchmark evidence contains no material regressions, Guardian creates a verification-only contract.

In this state:

  • no implementation change is authorized;
  • no test modification is authorized;
  • speculative cleanup and refactoring are prohibited;
  • Codex inspects and verifies the supplied evidence;
  • protected verification is still required;
  • human approval is still required;
  • the system reports that no code repair is needed.

This distinction prevents an AI agent from inventing work simply because it was given access to a repository.

Proven on a real pull request

I validated the full workflow using a real pull request in the project repository.

The pull request introduced an intentionally inefficient repeated lookup in PR-comment generation. Functional tests continued to pass, but the protected benchmark gate detected a serious performance regression:

  • Metric: pr_gate_generation_latency_ms
  • Regression: +135.21%
  • Severity: Critical
  • Release readiness: Needs Review
  • Score: 70/100

Guardian generated benchmark evidence, investigation guidance, a Codex handoff, validation commands, and protected finish conditions.

Codex traced the regression to a repeated linear lookup performed for every regressed metric, producing approximately quadratic behavior.

The focused repair:

  • removed the redundant lookup;
  • restored linear behavior;
  • preserved output content and ordering;
  • added deterministic regression coverage.

The protected workflow then produced fresh evidence:

  • Material regressions: 0
  • Release readiness: Ready
  • Score: 100/100

The benchmark threshold, metric directions, protected evaluator, provenance, and human approval requirement remained unchanged throughout the repair.

This demonstrates the project’s main value: Guardian can detect a performance problem that ordinary functional testing misses and guide a safe repair without allowing the repair process to weaken the guardrail.

Protected pull-request benchmark gate

The project includes a protected GitHub pull-request benchmark workflow.

It compares the exact protected base revision and pull-request head using:

  • the same GitHub-hosted runner;
  • separate restricted containers;
  • identical workload and measurement settings;
  • a benchmark harness copied from the protected base branch;
  • a trusted evaluator from the protected base branch;
  • deterministic benchmark evidence and provenance.

The untrusted pull-request code cannot modify the evaluator, threshold, metric-direction policy, protected baseline, or workflow definition used to judge it.

A separate trusted publisher validates the workflow and pull-request identity before posting or updating the benchmark decision on GitHub. The publisher never executes pull-request code.

This separation is important because an automated performance gate is only trustworthy when the code being evaluated cannot redefine the rules used to evaluate itself.

Codex Handoff Pack

Guardian generates a complete set of artifacts that developers can inspect, assign, download, or use in another workflow.

Depending on the interface and whether a repair is required, the Handoff Pack includes equivalent forms of:

  • the Codex repair goal;
  • the Repair Contract in Markdown and JSON;
  • benchmark reports in Markdown and HTML;
  • release-readiness guidance;
  • regression triage;
  • a GitHub issue handoff;
  • a generated CI workflow;
  • pull-request decision content;
  • trusted gate summaries.

When a regression exists, Guardian exposes the bounded repair workflow.

When no repair is required, speculative implementation instructions are hidden and only verification-safe guidance is presented.

Every representation is generated deterministically from the same Python evidence and policy.

How I built it

The project uses:

  • Python 3.12 for benchmark analysis and repair policy;
  • Typer for the command-line interface;
  • Rich for readable terminal output;
  • FastAPI for the production analysis API;
  • Next.js 15 and TypeScript for the production frontend;
  • Tailwind CSS for responsive presentation;
  • Streamlit for an additional local Python-first interface;
  • Pytest for automated testing;
  • Ruff for linting and formatting;
  • GitHub Actions and Docker for protected pull-request evaluation;
  • deterministic Markdown, HTML, JSON, and YAML generation.

The project currently includes 162 automated Python tests, along with frontend linting, TypeScript checks, production builds, workflow validation, protected benchmark checks, and manual production verification.

Judges and developers can start with the live dashboard or run the project locally:

git clone https://github.com/OmprakashSahani/codex-benchmark-guardian.git
cd codex-benchmark-guardian

pip install -e ".[dev,dashboard]"
npm install

make lint
make format-check
make test

npm run lint
npm run typecheck
npm run build

The project supports modern desktop and mobile browsers for the hosted dashboard. The local Python tools require Python 3.12 or newer, while the protected repository gate runs through GitHub Actions and Docker on Ubuntu runners.

How I used Codex and GPT-5.6

I used Codex with GPT-5.6 throughout the project as an engineering collaborator.

Codex helped with:

  • implementation planning;
  • focused code changes;
  • test generation;
  • debugging;
  • CLI and API development;
  • frontend integration;
  • workflow hardening;
  • documentation;
  • pull-request review;
  • security and edge-case analysis.

Codex contributed to features including:

  • multi-metric benchmark comparison;
  • higher-is-worse and lower-is-worse metric handling;
  • deterministic readiness scoring;
  • regression triage;
  • Markdown and HTML reporting;
  • CI failure behavior;
  • the Handoff Pack;
  • GitHub issue and workflow generation;
  • the protected PR benchmark gate;
  • FastAPI and Next.js integration;
  • the Repair Contract;
  • repair-required and verification-only Codex Goals.

Codex reviews also found meaningful issues that were corrected before merge.

For example:

  • verification-only contracts initially inherited actions that could authorize implementation and test changes;
  • verification-only Codex Goals initially ended with an unconditional repair-loop instruction;
  • fresh protected verification evidence needed to be distinguished from immutable original failing evidence;
  • a legacy speculative fix prompt remained accessible after switching to a verification-only result;
  • stale artifact selection could preserve content from a previous dashboard scenario;
  • protected workflow paths and read-only container behavior required focused corrections.

These findings improved correctness, security, compatibility, and the safety of the agent workflow.

I retained responsibility for:

  • defining the product problem;
  • selecting the architecture;
  • designing the readiness policy;
  • establishing the protected evidence model;
  • deciding the Codex safety boundaries;
  • reviewing and accepting changes;
  • validating production behavior;
  • approving and merging pull requests.

Codex assisted with implementation and review. It did not define the benchmark policy, approve its own work, declare a repair successful, deploy the application, or merge changes.

Challenges

Separating real regressions from benchmark noise

Performance measurements can fluctuate because of runner noise and environmental variation.

I addressed this through paired base-versus-head measurements on the same runner, consistent benchmark settings, explicit thresholds, deterministic evaluation, and preserved provenance.

Protecting the evaluator from untrusted code

A pull request should not be able to change the workflow, harness, or evaluator that decides whether it passes.

Designing a secure boundary between protected base code, untrusted pull-request code, and a write-capable GitHub publisher required careful workflow architecture, isolated containers, restricted permissions, and several rounds of testing and review.

Supporting different metric meanings

An increase is harmful for latency but beneficial for throughput.

The system therefore supports per-metric directions instead of applying one assumption to every benchmark value.

Designing safe Codex behavior

A prompt that simply says “fix the benchmark” could encourage threshold changes, evidence manipulation, test weakening, overfitting, or unnecessary edits.

The Repair Contract required a more explicit model of:

  • allowed actions;
  • prohibited changes;
  • trusted evidence;
  • completion criteria;
  • stop conditions;
  • human authority.

Keeping every interface consistent

The CLI, Streamlit app, FastAPI service, Next.js dashboard, GitHub workflows, reports, and generated artifacts all needed to use the same underlying policy.

I kept the Python engine as the canonical source of truth and avoided duplicating decision logic in the frontend.

What I learned

This project taught me that adding an AI agent to a developer workflow is not only a prompting problem. It is also a systems, security, evidence, and authority-design problem.

I learned how to:

  • design deterministic benchmark comparisons;
  • model metric direction and severity correctly;
  • create CI-friendly release decisions;
  • build trusted base-versus-head performance evaluation;
  • preserve benchmark provenance;
  • separate untrusted execution from trusted publication;
  • turn benchmark evidence into bounded agent instructions;
  • distinguish repair from verification;
  • design for human approval rather than uncontrolled automation;
  • use Codex iteratively for implementation, debugging, testing, and review.

Most importantly, I learned that trustworthy agentic development requires more than asking an AI to produce a patch.

The workflow must define what the agent is allowed to change, preserve the evidence used to judge it, verify the result independently, and keep final authority with a human.

What’s next

The next stage would focus on making Guardian easier to adopt across more engineering stacks.

Potential improvements include:

  • adapters for pytest-benchmark, ASV, JMH, k6, Locust, Lighthouse, and ML evaluation outputs;
  • historical benchmark storage and trend visualization;
  • repository-specific readiness policies;
  • organization-level dashboards;
  • GitHub App installation and configuration;
  • richer measurement-noise analysis;
  • ownership routing and notifications for affected components.

The core principle will remain unchanged:

Codex can help repair performance regressions, but trusted evidence and human approval must determine whether a change is safe to merge.

Built With

  • benchmarking
  • ci/cd
  • codex
  • developer-tools
  • devops
  • docker
  • fastapi
  • github
  • github-actions
  • gpt-5.6
  • next.js
  • node.js
  • openai
  • performance-testing
  • pytest
  • python
  • react
  • rest-api
  • rich
  • ruff
  • tailwind-css
  • test-automation
  • typer
  • typescript
  • vercel
Share this project:

Updates