SigMap Codex Bridge

Measure whether ranked repository context changes Codex outcomes.

Comment

Inspiration

Repository-context tools are easy to demo and difficult to evaluate. A successful agent run does not tell us whether retrieved context helped, whether the model would have succeeded without it, or whether the comparison was fair.

SigMap Codex Bridge was built to turn that question into a reproducible experiment. Its goal is not to prove that SigMap always wins. Its goal is to preserve enough evidence to show when ranked context helps, when it does not, and why the result can be trusted.

What it does

SigMap Codex Bridge is a paired A/B benchmark and evidence layer for Codex. It runs the same repository task in two conditions:

  • Raw: Codex receives the task without SigMap context.
  • SigMap: Codex receives ranked repository context generated by SigMap.

Both conditions use the same pinned Git revision and benchmark contract. Every attempt runs in its own detached worktree. Correctness is scored from tests and observable outputs, not from the context SigMap retrieved. The bridge retains raw attempt artifacts and regenerates deterministic comparison reports.

The shipped CLI also provides preflight validation, resumable execution, explicit cost limits, paired regression gates, portable replication packs, live-readiness diagnostics, submission-provenance validation, and a zero-credit historical replay.

Measured result

The checked-in Build Week experiment was run on 2026-07-18. It contains three tasks, two conditions, and three repetitions per condition: 18 retained attempts in total.

Measure Raw SigMap Observation
Candidate test and static-check suites 9/9 passed 9/9 passed No correctness advantage observed
Median runtime 249.089 s 186.590 s 25.1% lower with SigMap
Median total input 766,538 tokens 562,358 tokens 26.6% lower with SigMap
Median command events 20 15 25% lower with SigMap
Median output 7,169 tokens 4,587 tokens 36% lower with SigMap

This is a small historical sample, not a general performance claim. One task used more input with SigMap: artifact-run-status recorded a median of 534,456 raw input tokens and 606,285 with SigMap. That negative result remains visible in the report. The sample is too small for a statistical-significance claim.

The frozen report is tied to Git revision 504e823853459fef0c66e0b02915b4fc59ec3151, report commit d7c9877906af083ae0724e50175f859386a52e7b, and SHA-256 689698a2525cb77142a3aae33295ef6d7de18a6d0bb848c85546f70c61acf490.

How it was built

SigMap Codex Bridge architecture

  1. Validate the task. A strict versioned YAML or JSON contract defines the repository, pinned revision, prompt, allowed files and symbols, safe argument-array commands, tests, and limits. Shell command strings are rejected.
  2. Preflight the source. The bridge rejects missing revisions, dirty sources, unavailable commands, setup failures, and failing baseline tests before spending model credits.
  3. Isolate every attempt. A detached Git worktree gives each run a clean, recoverable workspace. Condition order alternates to reduce order bias.
  4. Deliver or withhold ranked context. The SigMap condition fails closed if context retrieval is missing, empty, timed out, or unsuccessful. It never silently becomes a raw run.
  5. Run Codex under limits. The runner records command and tool events, runtime, token use, changed paths, and terminal state. Interrupted batches can resume.
  6. Score independently. Tests, static checks, observable outputs, file and symbol precision/recall, unexpected files, and patch size are evaluated without a SigMap context ground-truth field.
  7. Retain and verify evidence. Attempt artifacts are written atomically. A SHA-256 audit chain and checkpoint expose missing or altered records. Reports can be regenerated from retained artifacts.

How Codex was used

Codex was the implementation partner for the bridge itself, not merely the workload being measured. It helped build the package and schemas, worktree isolation, audit log, benchmark contracts, independent scorers, paired runner, resume and recovery paths, retained evidence, replay CLI, diagnostics, documentation, and submission gates. It also ran the benchmark attempts whose 18 unique thread IDs remain in the artifacts.

Important decisions made with Codex included:

  • Pin raw and SigMap conditions to the same revision and configuration.
  • Treat missing or invalid SigMap context as a failed attempt, never as raw.
  • Determine correctness from tests and observable outputs, not retrieved context.
  • Preserve every attempted run, including failures and the negative task result.
  • Alternate paired condition order and retain the pairing in reports.
  • Label historical replay as zero-credit and never present it as a fresh model run.
  • Keep the deterministic orchestration and evidence layer outside the stochastic agent execution.

Precise GPT-5.6 contribution

In Codex session 019f75cb-5dfc-7f03-a9c1-94f86dd92c8c, GPT-5.6 added structured, fail-closed submission-provenance validation. The validator cross-checks the /feedback UUID, requires the GPT-5.6 model label, records a concrete contribution and safe argument-array verification command, and rejects missing or repository-escaping changed-file paths. Contract tests cover mismatched models and sessions, unsafe command strings, missing files, and escaped paths.

The same GPT-5.6 session synchronized the README and Devpost evidence, prepared the submission metadata, and produced the under-three-minute demo plan.

How it looks

Strict benchmark contracts

Benchmark task validation in the CLI

benchmark validate exposes the exact prompt, paths, commands, and constraints that will govern a run before any model call begins.

Zero-credit judge replay

Zero-credit historical replay in the CLI

sigmap-bridge demo verifies and replays the frozen evidence locally. It makes no Codex, SigMap, Git, or network calls and consumes no model credits.

Machine-readable evidence

Machine-readable replay output

The same replay is available as JSON for inspection or automation.

Readiness diagnostics

Live and replay readiness diagnostics

doctor separates zero-credit replay readiness from optional live-run requirements, including the Python version, Git, SigMap, Codex, and authentication.

Submission evidence validation

Submission metadata and provenance validation

submission validate checks local evidence integrity and external submission metadata. The screenshot deliberately shows the fail-closed missing-video gate that was active before the public demo URL was added.

Challenges

  • A Homebrew Codex launcher failure initially looked like missing authentication. The diagnostics now distinguish executable launch failures from auth failures.
  • The machine's Apple Python 3.9 was outside the supported range. The clean install and wheel checks were repeated with supported Python.
  • Early SigMap delivery generated context but did not read the generated payload into the run. The execution contract was corrected so delivered context is auditable.
  • Git worktree creation was blocked once by sandbox permissions. The run was repeated with scoped permission and the recoverable lease was explicitly cleaned.
  • A child npx sigmap process timed out. The bridge records this as a context failure instead of silently falling back to the raw condition.
  • Small samples invite overclaiming. The report therefore keeps per-task values, uncertainty, and the negative result alongside the aggregate medians.

Accomplishments that we're proud of

  • 18 retained attempts across 18 unique Codex threads: nine raw and nine SigMap.
  • 18/18 bridge-owned worktrees reported cleaned.
  • Both conditions passed all 9/9 candidate test and static-check suites.
  • The zero-credit replay is byte-linked to the frozen report checksum.
  • The repository has 99 automated tests plus one opt-in live CLI smoke test that is skipped by default.
  • CI covers macOS and Linux across CPython 3.10 through 3.14.
  • The scorer contract deliberately contains no SigMap context ground truth.

What we learned

Context quality should be measured, not inferred from a convincing demo. Isolation and scoring were only half the problem; failed attempts, delivery failures, exact revisions, cleanup state, and raw event records also had to survive for the comparison to be reviewable.

We also learned that replay and fresh execution serve different purposes. Replay makes evaluation accessible to judges without credentials or credits, while retained provenance makes clear that the replay is historical evidence rather than a new model run.

How judges can test it

Supported platforms are macOS and Linux with CPython 3.10 through 3.14.

git clone https://github.com/manojmallick/sigmap-codex-bridge.git
cd sigmap-codex-bridge
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install .
sigmap-bridge demo
sigmap-bridge demo --json
sigmap-bridge submission validate submission/build-week-2026.json

The demo needs no Codex or SigMap authentication, model credits, test account, or network access. A live benchmark is optional and should begin with sigmap-bridge doctor --live and sigmap-bridge benchmark preflight.

Repository

Public source, tests, frozen artifacts, methodology, schemas, and MIT license:

https://github.com/manojmallick/sigmap-codex-bridge

What's next for SigMap Codex Bridge

  • Run a larger independent replication and publish fresh paired evidence.
  • Add provider adapters without weakening the benchmark and scorer contracts.
  • Strengthen signed provenance and artifact-verification threat modeling.
  • Build an aggregate comparison view for multiple benchmark packs.

These are roadmap items, not features claimed by the current release.

Built With

Share this project:

Updates