Auto-Research OverWatch (AROW)

Inspiration

Research is a loop: run an experiment, read the result, change the method, run again. Hand that loop to a team of AI agents and it has three ways to die — it runs away and never stops (burning tokens forever), it spins in place (churning with no real progress), or, the worst one, it looks like it closed cleanly but there's a lie inside it: an agent that claims it's done, quietly edits the data, or builds a conclusion on a poisoned input. That's not a crash — it's a confident lie, and nobody would ever catch it. Existing loop detection is linear, single-agent, and welded inside each vendor's CLI; it only knows how to stop, not diagnose, attribute, or keep receipts. The real problem in Loop Engineering isn't killing loops — it's putting a gauge and a brake on them.

What it does

AROW is an asynchronous, graph-native supervision layer for multi-agent research. You give it one research question; it plans the task graph automatically and a team of agents runs the experiments, each in its own sandbox. The supervisor watches the whole graph using hard signals only — file hashes, metric curves, exit codes, never an LLM judge — and catches five failure modes, each with an automatic response:

• Hung — restart from the last checkpoint, then fuse if it stalls again
• Plateau — early-stop, write the negative result, free the compute
• False completion — a gate re-runs the tests and bounces it
• Broken comparability — manifest four-tuple mismatch → block and blame the culprit
• Out-of-scope contamination — taint propagates along dependency edges

Responses climb a five-rung ladder — bounce, blame-routing, downstream invalidation, graph surgery, fuse — and every rung is written to a black-box log. One line: don't trust what the agent says; check whether the world-state actually changed.

How we built it

Pure Python, no heavy frameworks: asyncio for barrier-free ready-set scheduling with resource slots, networkx for the graph, a stdlib HTTP server plus a single-file hand-rolled SVG dashboard. The research-stage breakdown takes cues from Agent Laboratory; the in-node task format borrows from AIDE; the failure taxonomy is operationalized from Berkeley's work on why multi-agent LLM systems fail; and the execution semantics — dependencies gate readiness, incremental invalidation, taint that spreads along edges — come straight from the async build-system lineage (Bazel/make). Workers are a lightweight bash-only agent (allowlist + step budget) wired through OpenRouter, so any model plugs in — in the demo the same net supervised GLM and GPT workers. The supervision layer itself (five-rung ladder, budget fuse, plateau early-stop, blame-routing, comparability gate) has no off-the-shelf equivalent — that's the part that's ours.

Challenges we ran into

The hardest one was honest: how do you prove "catches fabrication" is real and not staged? So we ran an adversarial test — we fed a training node a misleading instruction telling it to skip the official tool and hand-write its own result provenance. GLM-5.2 was too conservative to even produce the forged manifest — its bad work never reached the gate. But GLM-4.5-air, capable enough to understand and execute that instruction, took the bait: it really did fabricate the data hash — and got caught at the comparability gate on a hash mismatch, blamed, and its result withheld. The other recurring one was worker budget discipline: a live agent would burn through its step budget, the system degraded per its own rules, we found the root cause, fixed the task spec, and the rerun went green — the whole "fuse → honest fallback → fix → rerun" philosophy, which we ended up running on ourselves during the build.

Accomplishments that we're proud of

Empty repo to demoable overnight: a full async graph executor plus supervision layer, 8 deterministic and replayable failure scenarios, a single-file self-explaining dashboard (with an evidence drawer — click an incident card and see the culprit's and baseline's hashes side by side), and a one-click demo. On the live side we ran real research: one sentence in, real data, a real GBDT training node (real curve, real checkpoints), real agents, real acceptance gates — a conclusion in two minutes, reproduced twice, at under one cent of live API spend for the entire day. The best single frame is that fabrication catch: it proves the product's core claim — the gate works on real agents too, and the more capable the agent, the more it needs the gate.

What we learned

A model should do exactly two things — plan and work; the discipline has to be deterministic code. The moment you hand "is this correct?" to the model, a weaker model means weaker order. The counterintuitive lesson: a stronger agent isn't a safer one — it's precisely the model capable enough to follow a shortcut but not judgmental enough to refuse it that falls into the fabrication trap, which makes the supervisor not optional but necessary. And success on a sunny day proves nothing — the value shows up when the system is misled and pressure-tested and still holds the line.

What's next for AROW (Auto-Research OverWatch)

• Free-topology generation — today the planner adapts the research question onto a validated template graph, a deliberate reliability choice; letting the planner build topology freely and wiring it into the executor is the next step.
• A drop-in proxy form factor — an OpenAI-compatible proxy so any agent framework attaches supervision by changing one base_url.
• Real runtime — swap the long node's simulated compute for a real training backend and upgrade the bash sandbox to docker/remote runtimes.
• Beyond research — take "give an agent team a PI" anywhere empirical work is handed to a team of agents.

Built With

• pyton