Inspiration

Most AI benchmarks pause the environment while a model thinks. Real agents do not get that luxury: robots keep moving, tools continue running, and the world changes while an agent is deliberating.

I wanted to test one question:

Can an AI agent account for the fact that the world keeps moving while it thinks?

That idea became DelibraShift.

What it does

DelibraShift is a diagnostic arena for AI agents operating in a non-pausing simulated world.

An agent controls a craft affected by gravity, wind, inertia, and deadlines. It observes the world, but its next action becomes active only after a fixed deliberation delay. An action that was correct when the observation was received may already be wrong when it is executed.

Instead of returning only a leaderboard score, DelibraShift diagnoses several capabilities:

• world-model and prediction fidelity,
• temporal anticipation,
• use of feedback,
• final outcome,
• action and formatting reliability.

The goal is not only to show that an agent failed, but to explain why it failed.

How we built it

I provided the original concept, objectives, and operating constraints. Codex acted as the primary implementation and delivery agent, while other frontier models were used for review, adversarial testing, and cross-checking.

Codex worked across the complete development process: repository exploration, architecture, implementation, testing, failure diagnosis, recovery, Git and pull-request workflows, reporting, documentation, and video production.

DelibraShift uses:

• deterministic simulated physics,
• frozen and versioned prompts,
• canonical JSONL logs,
• matched experimental arms,
• repeated runs and control probes,
• atomic writes and checkpoints,
• resumable experiments,
• offline deterministic replay.

The first complete experiment produced 84 canonical log files, including 78 full episodes and 6 registered control probes, across 3 repetitions and 21 matched experimental cells.

Challenges we ran into

The largest challenge was separating agent cognition from unrelated effects such as API latency, hardware speed, formatting failures, and lucky trajectories.

Wall-clock time is recorded only as telemetry. The scored delay is represented using deterministic simulated ticks, so results are not determined by the speed of the computer or model endpoint.

Another challenge was making long-running experiments resilient. During the first full run, an interruption occurred after many episodes had completed. The recovery system identified the first missing episode and resumed from that point without regenerating or overwriting previous results.

We also had to distinguish between a model failing to reason and a model merely failing to produce the expected output format.

Accomplishments that we're proud of

• A complete 84-log controlled experiment.
• Deterministic offline replay requiring zero API keys.
• Recovery from an interrupted run without losing completed work.
• Diagnostic results instead of a single opaque score.
• Frozen prompts, explicit support counts, and traceable evidence.
• A reproducible report and demonstration video produced through Codex.
• A workflow that moved from the initial idea to a working, tested project in approximately one day.

What we learned

The main lesson was that effective agent work requires more than a single prompt.

An agent needs:

• a clear final objective,
• visible intermediate state,
• explicit completion criteria,
• permission to inspect and repair failures,
• quality gates before declaring success.

With those conditions, Codex maintained the main goal across architecture, implementation, testing, recovery, reporting, repository preparation, and final video production.

We also learned that a final success rate alone is not enough. Two agents can achieve a similar outcome for completely different reasons. Diagnostic axes make those differences visible and provide information that can be used to improve an agent architecture.

What's next for DelibraShift

The next steps include:

• additional models and agent architectures,
• noisy and incomplete observations,
• uncertain forecasts,
• dedicated replanning and perception diagnostics,
• additional real-time environments,
• multi-agent scenarios,
• easier Basic and Laboratory interface modes,
• public, reproducible experiment reports.

The long-term goal is to make DelibraShift an open laboratory for determining which models and agent architectures are suitable for systems that must act while the world keeps moving.

Built With

• ffmpeg
• github-actions
• gpt-5.6
• html
• javascript
• jsonl
• openai-codex
• playwright
• python