Codex Time Machine

Replay coding agents at moments before known failures, reveal what they failed to investigate, and test whether minimal hindsight can change the engineering timeline.

Comment

Codex Time Machine

Git remembers what changed. Codex Time Machine reconstructs why.

Codex Time Machine is a forensic developer tool for testing a difficult counterfactual question:

Would an AI engineering agent have made a different decision if it had investigated the right evidence?

Instead of rewriting history or asking a model to explain its hidden reasoning, Codex Time Machine reconstructs only the agent’s observable engineering history: messages, commands, file operations, tests, outputs, and final decisions.

It then introduces the minimum future clue the agent originally lacked, replays the task from the same starting point, and compares the two histories using integrity-checked evidence.

Inspiration

AI coding agents can make technically plausible decisions that later turn out to be based on incomplete investigation.

Git can show us which files changed. Logs can show us which commands ran. Tests can show us what passed or failed. But none of these tools answer a deeper question:

Was the outcome caused by a bad implementation, or did the agent simply fail to investigate the right thing?

We wanted a way to examine an AI engineering decision without relying on post-hoc storytelling or hidden chain-of-thought.

That led to the core idea behind Codex Time Machine:

  1. Freeze an original historical run.
  2. Seal later evidence behind a temporal boundary.
  3. Identify the investigation the agent missed.
  4. Compress that knowledge into a minimal, non-answer-revealing clue.
  5. Replay the task from the same starting snapshot.
  6. Compare the observable histories.
  7. Measure whether the missing investigation was actually covered.

The project is designed to make AI engineering decisions falsifiable, auditable, and reproducible.

What it does

Codex Time Machine turns a historical AI coding session into a controlled counterfactual experiment.

The accepted demo analyzes a LegalRAG retrieval decision. The original run selected a production-default retrieval strategy after evaluating only a narrow set of examples.

Later evidence showed that the evaluation was not representative enough.

Codex Time Machine then:

  • reconstructs the original observable trajectory;
  • seals later evidence behind a future-information boundary;
  • identifies the missing representative evaluation;
  • generates a minimum Ghost Engineer clue;
  • replays the task from the same isolated starting snapshot;
  • aligns the baseline and replay histories;
  • identifies their first observable divergences;
  • measures target-specific coverage;
  • generates a Counterfactual Reasoning Receipt.

The minimum clue used in the demo is:

Before selecting a production retrieval default, verify the recommendation on a representative labeled query set covering multiple query types.

The clue does not reveal:

  • benchmark answers;
  • metric values;
  • the preferred retrieval strategy;
  • whether the original recommendation was correct;
  • which solution should win.

It only exposes the missing investigation.

The controlled result

The original run and replay produced the following target-coverage result:

  • Baseline: MISSED
  • Replay: OBSERVED
  • Target-level direction: INCREASED
  • Aggregate shift: TARGET_COVERAGE_INCREASED
  • Total activity: LESS_TOTAL_ACTIVITY
  • Target relationship: INCREASED_TARGET_COVERAGE

The most important result is not that the replay performed more work.

It actually performed less total observable evaluation activity than the baseline.

However, it covered the specific representative-evaluation target that the baseline missed.

This distinction became one of the central ideas of the project:

More activity is not necessarily better investigation. What matters is whether the agent covered the right missing target.

Codex Time Machine does not claim that the replay was technically correct, that retrieval quality improved, or that the intervention caused every difference. It only reports what changed observably and which investigation target was covered.

How we built it

The backend is implemented in Python using Pydantic and Pytest.

It is organized as a nine-phase evidence pipeline:

  1. Temporal Information Boundary
    Defines which information existed during the original run and which information became available later.

  2. Isolated Historical Run
    Executes or imports the baseline run inside a controlled workspace snapshot.

  3. Observable Trajectory
    Normalizes messages, commands, file activity, tests, outputs, and completion events.

  4. Known-Future Context
    Packages later evidence without allowing it to leak into the historical run.

  5. Blind-Spot Assessment
    Determines which investigation, assumption, constraint, or experiment was missing.

  6. Ghost Engineer Intervention
    Produces the smallest future clue that communicates the missing investigation without revealing the answer.

  7. Controlled Counterfactual Replay
    Replays the task from the same starting snapshot with the approved clue.

  8. Observable History Divergence
    Aligns baseline and replay event histories and identifies structural and investigative differences.

  9. Counterfactual Target Coverage
    Determines whether the replay observably addressed the specific target the baseline missed.

Every phase publishes deterministic JSON and Markdown artifacts with SHA-256 manifests.

Later phases validate earlier artifacts through:

  • canonical hashes;
  • source-lineage hashes;
  • immutable phase receipts;
  • protected output directories;
  • event-reference validation;
  • fail-closed integrity checks.

The accepted backend contains more than 500 passing tests.

The frontend is a static React, TypeScript, and Vite application with 19 focused UI tests.

It includes:

  • Mission Control;
  • a nine-phase evidence rail;
  • Guided Demo and Judge Mode;
  • a temporal evidence boundary;
  • the Ghost Engineer clue;
  • baseline-versus-replay comparison;
  • divergence highlights;
  • target-coverage results;
  • a technical evidence drawer;
  • a downloadable Counterfactual Reasoning Receipt.

The public demo loads a sanitized accepted evidence bundle and requires no API key, backend server, or live model.

How Codex and GPT-5.6 were used

Codex was the primary engineering agent used throughout OpenAI Build Week.

It accelerated:

  • repository-wide architecture;
  • Python implementation;
  • schema design;
  • test generation;
  • failure reproduction;
  • integrity validation;
  • frontend development;
  • accessibility work;
  • documentation;
  • deployment preparation.

GPT-5.6 was used through Codex for complex repository-wide implementation and for the historical controlled run.

The human decisions remained central. We defined:

  • the temporal evidence boundary;
  • the rule that future answers must not leak into replay;
  • the minimum-clue constraint;
  • the distinction between activity volume and target coverage;
  • the requirement that every published conclusion resolve to observable evidence;
  • the prohibition on technical-correctness and causality claims.

Routine regression testing used deterministic fake providers and recorded fixtures to avoid unnecessary live-model calls.

The final public demo invokes no live model and consumes no Codex credits.

Challenges we faced

Preventing future-information leakage

The hardest problem was not generating a replay. It was deciding what the replay was allowed to know.

If the replay receives the benchmark answer, preferred solution, or winning metric, the experiment becomes meaningless.

The intervention therefore had to reveal the missing investigation without revealing its result.

Avoiding hidden-reasoning claims

We deliberately avoided reconstructing private chain-of-thought.

The system operates only on observable events and clearly states the limits of lexical and structural analysis.

Separating “more work” from “the right work”

Early versions risked treating additional commands or files as evidence of improvement.

We redesigned the assessment so that total activity and target-specific coverage are separate dimensions.

Building trustworthy cross-phase evidence

Every later phase depends on earlier artifacts.

We implemented canonical hashing, source-lineage validation, phase receipts, atomic publication, overwrite protection, and semantic revalidation to prevent modified evidence from silently becoming accepted.

Making a complex backend understandable

The final challenge was presentation.

The frontend had to explain a nine-phase forensic pipeline in less than three minutes without reducing it to a generic dashboard.

Judge Mode presents the experiment as an eight-step narrative:

  1. Original decision
  2. Later evidence
  3. Missing investigation
  4. Minimum Ghost clue
  5. Controlled replay
  6. First divergence
  7. Target coverage
  8. Integrity receipt

What we learned

We learned that evaluating AI engineering agents requires more than checking whether the final answer was correct.

A useful evaluation system must ask:

  • What evidence was available?
  • What investigation was attempted?
  • What was never tested?
  • What minimum information would have changed the observable process?
  • Did the replay cover the missing target?
  • Can every conclusion be traced to accepted evidence?

We also learned that a smaller replay can be more informative than a larger one.

In the accepted experiment, the replay performed fewer total observable actions, yet addressed the exact investigation the baseline lacked.

Accomplishments

  • Built a complete nine-phase temporal evidence pipeline.
  • Reconstructed AI engineering histories without exposing hidden chain-of-thought.
  • Created a minimum non-answer-revealing Ghost Engineer intervention.
  • Built deterministic counterfactual replay from an identical starting snapshot.
  • Demonstrated a grounded MISSED → OBSERVED target-coverage shift.
  • Separated total activity from target-specific coverage.
  • Added canonical manifests, phase receipts, semantic validation, and atomic publication.
  • Created more than 500 backend tests and 19 frontend tests.
  • Built a static, public, judge-ready demo requiring no API key or live model.
  • Generated a downloadable Counterfactual Reasoning Receipt.

What’s next

The next version could support:

  • importing arbitrary Codex repositories and historical runs;
  • comparing multiple possible interventions;
  • additional target-specific coverage policies;
  • team-level engineering-decision audits;
  • CI and pull-request integrations;
  • signed external receipt storage;
  • stronger container-level replay isolation;
  • comparisons across agents and model versions.

The long-term goal is to make counterfactual evaluation a normal part of AI-assisted software engineering.

Links

Codex Time Machine does not rewrite history. It makes AI engineering decisions falsifiable.

Built With

Share this project:

Updates