CodexForge: Built by Codex for Codex

A Codex-built deterministic orchestration system that runs long-lived parallel agents to solve, validate, and optimize 400 Kaggle NeuroGolf ARC-AGI-1 tasks into verified ONNX programs.

Comment

Inspiration

The Kaggle NeuroGolf Championship presents 400 ARC-AGI-1 visual-reasoning tasks. Each transformation must be inferred from a few examples, implemented as an executable ONNX graph, validated for correctness, and optimized for memory and parameter cost.

Solving one task is difficult. Coordinating hundreds of long-running Codex sessions over days or weeks creates a second problem: a conversational parent agent should not remain active merely to poll workers and preserve campaign state.

CodexForge was built around one principle:

Use Codex for reasoning; use deterministic software for orchestration.

What it does

CodexForge is a persistent Python control plane for parallel GPT-5.6 Sol workers.

Each isolated Codex worker:

  • Inspects one task’s examples and ARC-GEN generator
  • Infers the underlying transformation
  • Builds an executable ONNX program
  • Tests and validates the result
  • Measures its official NeuroGolf cost
  • Optimizes the graph without sacrificing correctness
  • Saves a reproducible builder, explanation, logs, and submission model

The deterministic parent handles:

  • Task selection and bounded concurrency
  • Process supervision and task locking
  • Timeouts, cancellation, and rate-limit-aware retries
  • Structured logs and result summaries
  • Artifact verification
  • Skipping completed work
  • Safe recovery and resumability

This allows a campaign to run unattended without keeping another reasoning-model session continuously alive.

How I built it

CodexForge was designed and implemented through OpenAI Codex using GPT-5.6 Sol.

I defined the product direction, competition constraints, concurrency model, task isolation, validation requirements, artifact contracts, quota-safety rules, and acceptance criteria. Codex implemented the runner, quota supervisor, logging, retries, process cleanup, task-specific ONNX builders, tests, optimization passes, and documentation.

The system contains two main tools:

  • run_codex_tasks.py — the deterministic control plane that launches and supervises one Codex CLI process per task
  • codex_quota_supervisor.py — an optional supervisor for observing usage and safely handling explicitly authorized reset credits

The control plane and workers remain separate: Python provides durable orchestration, while Codex performs the open-ended reasoning and program synthesis.

Challenges I ran into

Long-running agent supervision

Workers may run for hours, fail, time out, or require later repair and optimization passes. Persisting state in files, logs, locks, and summaries makes the campaign recoverable without relying on conversational memory.

Parallel isolation

Each worker needs its own task scope. Task-local workspaces and advisory locks prevent concurrent agents from corrupting the same output.

Correctness versus optimization

NeuroGolf rewards small graphs, but correctness comes first. Workers validate candidates before reducing operations, parameters, or intermediate memory.

Distinguishing solver failures from quota failures

The runner records structured output and applies bounded retries only when captured evidence indicates a rate-limit condition.

Experimental Codex interfaces

Usage and reset-credit methods may change between Codex CLI versions, so that logic is isolated, explicitly opt-in, and documented as experimental.

Accomplishments that I’m proud of

Codex did not only write the orchestration system—the system launches Codex to create further executable programs.

Each worker can independently infer a visual algorithm, select suitable ONNX operations, build and test a graph, optimize it, and preserve reproducible evidence.

Existing solutions include compact graphs using operations such as Einsum, TopK, Gather, Slice, Where, ConvTranspose, quantized convolution, etc. Some tasks were also checked against exhaustive generator-derived cases rather than only the visible examples.

I am especially proud of the separation between intelligence and reliability:

  • Codex handles interpretation, synthesis, debugging, and optimization.
  • Deterministic Python handles scheduling, persistence, recovery, and verification.

What I learned

Agent orchestration works best when every worker has:

  • One bounded assignment
  • An isolated workspace
  • Explicit sources of truth
  • A deterministic validation gate
  • A measurable objective
  • A required artifact contract

I also learned that long-running agent work should be managed like any distributed workload—with observability, retries, cancellation, locking, state persistence, and reproducible outputs.

What’s next for CodexForge

The next step is to generalize CodexForge into a reusable control plane for independently verifiable Codex workloads.

Planned improvements include:

  • A live terminal dashboard
  • Replay and judge-friendly demo modes
  • Automatic repair and optimization passes
  • A champion registry for the best task solutions
  • Regression and fuzz-testing gates
  • Dynamic worker scheduling
  • Runtime, cost, and success analytics

The same architecture could support repository migrations, vulnerability remediation, model compression, compiler optimization, test-driven repair, and other campaigns containing many isolated tasks.

Built by Codex for Codex.

Built With

  • chatgpt-5.6-sol
  • codex
  • codex-cli
  • onnx
  • python
Share this project:

Updates