One More Run
Most coding-agent loops treat a plausible diff as progress. ML does not care how plausible the diff looks. A cleaner architecture can still lose on held-out data, and a clever agent can repeat yesterday's failed idea if the evidence disappears with the session.
One More Run gives autonomous ML research two nested loops and one empirical rule:
An agent can change the program. Only an experiment can change the champion.
## What it does
The inner loop is cheap and exploratory. Codex reads the research objective, candidate contract, current champion, measured history, and relevant long-term memory. It can revise the complete Python training program—features, model architecture, loss, optimizer, schedule, and module boundaries—across a bounded number of turns. Static checks run locally. Codex explicitly marks the candidate ready before any GPU time is spent.
The outer loop is expensive and empirical. One More Run deploys a dedicated Akash GPU worker and
evaluates a content-addressed source bundle against fixed held-out validation targets. Direct
bearer-authenticated Akash is the proven default; protected mode routes the same request through
Pomerium Zero. The worker returns the candidate hash, evaluator identity, metric, duration, and
any failure. The controller verifies that receipt and makes one of three decisions: KEEP,
REJECT, or CRASH.
Only a measured improvement becomes the next champion.
Hindsight memory
recall ▲ │ retain verified evidence
│ ▼
objective ──► [ INNER: Codex inspect → edit → check → ready? ]
│ candidate source + SHA-256
▼
[ OUTER: Akash GPU fixed evaluation ]
│ metric + evaluator receipt
▼
KEEP / REJECT / CRASH
│
└──────────────► next outer iteration
## Long-term research memory
An append-only JSONL ledger remains the source of truth. After a receipt is validated and durably recorded, One More Run retains the objective, hypothesis, candidate identity, evaluator, metric, decision, and provider in a Hindsight memory bank.
Hindsight organizes memories across entities, relationships, and time. Its semantic, keyword, graph, and temporal retrieval paths bring relevant wins and failures back into the next inner loop. The agent does not merely remember the last message. It can reuse evidence from previous campaigns without allowing a memory outage to consume an experiment budget.
## Secure remote measurement
The mutable research agent and fixed evaluator do not share authority.
- Codex receives only the candidate workspace and its bounded research context.
- The local controller owns credentials, budgets, deployment, receipts, and the champion.
- Akash supplies ephemeral GPU compute through an immutable deployment manifest.
- In protected mode, Pomerium Zero is the deployment's only public service. Its policy validates service-account identity before forwarding to the private worker.
- The worker separately verifies its bearer token, serializes experiments, and executes the candidate in a child process with a bounded payload, scrubbed environment, and hard timeout.
- Pomerium consumes its identity header; the worker sees only the application credential it owns.
Before spending in protected mode, the controller verifies the exact Pomerium route, private upstream, and attached policy. Cleanup independently restores the prior Zero cluster IP and closes the Akash deployment, including on failure.
LOCAL CONTROL PLANE AKASH DATA PLANE
Codex ──candidate──► One More Run ──HTTPS──► Pomerium Zero :443
│ identity policy
budgets + ledger │ private route
│ ▼
Hindsight memory worker :8080
fixed evaluator
## How we built it
The controller is a small Python package run with uv. Adapters communicate through a strict
JSONL event protocol, so the same ledger and terminal can supervise local experiments or remote
compute. Candidates are normalized and hashed with SHA-256. A finished event is accepted only
when its candidate and evaluator identities match the experiment plan.
The whole-program evaluator accepts at most 32 Python files and 256 KiB of source. It holds the validation targets and seed outside the candidate contract, runs one experiment at a time, and reports crashes as evidence rather than erasing them.
Akash deployment lifecycle calls are bounded by one campaign deadline and an explicit deposit/bid ceiling. Runtime credentials are injected only into the in-memory manifest. Pomerium is pinned by both source revision and immutable container digest. Hindsight is optional and fail-open; its index can be rebuilt from the durable ledger.
## Challenges we faced
The hardest part was not asking an agent to edit code. It was deciding which component gets to call a change “better.” Keeping mutation local and measurement remote gave the evaluator a stable identity and made regressions visible.
The second challenge was authority. Pomerium identity, worker authentication, Akash account access, Codex credentials, and memory credentials each needed a different path and lifetime. Combining them into one token would have made the demo simpler and the system weaker.
The third challenge was memory timing. Storing every thought would turn memory into a larger prompt log. One More Run retains evidence only after the candidate/evaluator receipt is verified and the ledger write is durable.
## What we learned
- Nested loops need different budgets. Inner edits are cheap; outer measurements spend GPU time and therefore need an explicit readiness gate.
- A failed candidate is useful evidence when its identity and failure are retained.
- An identity-aware proxy does not replace application authentication. The two layers protect different boundaries.
- Long-term memory should be a rebuildable index over durable evidence, not a second source of truth.
- Cleanup is part of the research algorithm. A loop that cannot release compute or restore routing is not autonomous.
## What's next
The same edit/evaluate/remember contract can move from the compact held-out regression task to larger open-source training systems and domain prediction repositories. The evaluator can change; the invariants do not: bounded mutation, fixed measurement, content-addressed evidence, explicit spend, and memory earned by results.
One More Prompt starts the idea. One More Run tests it.
Log in or sign up for Devpost to join the conversation.