Skeptic

Inspiration

Agents fail on real tools for a boring reason: the documentation is wrong, and nothing tells them. An API says page_size accepts up to 500 and silently returns 50. It says a 502 means your write was not processed, and the write already landed, so the retry creates a duplicate. It says an unknown filter field returns 400, and instead returns 200 with an empty list, so the agent confidently reports zero results.

Every agent framework treats the tool spec as ground truth. None of them check.

Track 1 asked for a learning loop about how an agent uses tools over time, and this is the version of that problem we actually hit: not "can it learn a task" but "can it learn that its instructions are wrong, and correct them itself". The recurring job we built it around is a vendor-invoice cash report, and the shape of that job is why the trust gap matters. Maximor's CFO benchmark puts it at 96% of finance leaders wanting AI on the grunt work against 14% who trust it end to end. An agent cannot be trusted with an ERP until it knows what that ERP actually does rather than what its docs claim.

What it does

skeptic reverse-engineers its own tools.

1. Goes looking. A recon sweep tests every promise the docs make. Not "run the task and see what breaks", which only learns when you get unlucky in a useful way, but "the docs claim X, check X". 31 calls, 15 seconds, 12 of 12 documented promises broken.
2. Forms rival explanations. An anomaly says the docs and reality disagree, not why. The reflector proposes competing hypotheses in one batched call, each with the observation that would separate it from the others.
3. Designs an experiment, from parameterised probe templates (boundary, timing, idempotency, ordering, header burst, flag discovery), and commits to what each hypothesis predicts before running it. A model shown a result will rationalise almost anything; a model on record can be caught.
4. Keeps falsifiable beliefs. Confidence is a beta-binomial posterior over "the docs are correct", seeded pro-doc because trusting the docs is the failure being modelled. It can move back, which makes unlearning a measurement rather than a heuristic.
5. Unlearns. Switch the behaviour off and the posterior reverts and the belief retires. Observed: 0.24, 0.53, 0.66, 0.73, retired, in three rounds, without being told the world had changed.
6. Gets attacked. An adversary on a different harness reviews every confirmed belief with an inverted brief: not "is this supported" but "design the experiment most likely to break it".
7. Publishes. Confirmed beliefs compile into TOOLS.md, a corrected spec pairing each doc claim with observed reality and the probe that settled it, plus executable guards so a fresh agent gets the benefit without learning anything.

Everything is scored against a hidden answer key, so "it discovered a hidden behaviour" is measured, not asserted.

How we built it

The lab. A tool whose docs lie in exactly 14 known ways, with ground truth in a file the agent never sees. Silent truncation, coercion, eventual consistency, cursor expiry, an undocumented flag, and an idempotency hazard where 5% of writes return 502 after committing. Matching is structural, on class, operation and parameter. No LLM judges whether two sentences agree, because a model grading its sibling is how a benchmark quietly becomes fiction.

Detection is deterministic. A contract layer holds the docs' checkable promises as data and compares them to what came back on the wire. No model in that path, so anomalies reproduce. The model's job starts after the anomaly is found: explaining it, not spotting it.

Two model tiers, which is the cost answer. The executor does the volume on GLM-4.7-Flash through TensorMux; the reflector is invoked a handful of times per run for the thinking that matters. Batching all anomalies into one reflector call took that step from twelve slow round trips to one: 74 seconds, 12 anomalies, 24 rival hypotheses. Having a fast tier underneath also meant we could afford to throw away entire experiments once they turned out to be unsound, which this project ended up needing to do more than once.

AO is the control plane and the CLI is the measurement. ao/ORCHESTRATOR.md is a skill file that reads belief counts, run history and probe records, then picks the next action from recon, settle, apply, falsify, export, ab and bench. It never reimplements the loop, because a planner making judgement calls is fine but a planner that could reorder the measurement would make make bench SEED=n meaningless.

The adversary is the part AO made cheap. ao/falsify.py spawns it with ao spawn --harness claude-code while the proposer runs on GLM, then reads the verdict back from the worker's own worktree. Running a critic on a different harness from the proposer is a one-line change here and a week of adapter work without it, and that asymmetry is the entire reason it caught anything. Around it, ao/eventd.py subscribes to the daemon's SSE stream at /api/v1/events with Last-Event-ID replay and injects lifecycle events into the orchestrator session via ao send; isolated worktrees let probes, the adversary and the build run concurrently without touching each other's state; and ao preview serves the three-panel UI inside AO's browser panel. Two things we learned the hard way: ao spawn --prompt rejects a long brief, so the adversary's brief travels as a committed file, and --permission accept-edits is necessary rather than cosmetic, because a worker will sit at needs_input indefinitely without it.

We did not use Neatlogs; the loop writes its own structured run logs and probe records because the bench has to read them to compute precision and recall, so a second trace would have been redundant. AI Grants India offered GPT-5 Nano credits and the reflector is written to take a second endpoint, but the access did not arrive in the window, so both roles resolve to GLM and the README says so. There is no payment surface here, so Dodo Payments was not integrated rather than bolted on.

Challenges we ran into

One bug appeared seven times in seven disguises, and it is the most interesting thing we found: a measurement allowed to report something the evidence did not license.

A smoke test asserted page_size <= 50 against a 25-row store, so the cap was never exercised. A detector test asserted on anomaly kind, so a cursor check passed on an anomaly from elsewhere in the run. A probe swept a vendor with zero rows, got 0 for every page size, and read that as refuting a belief that was true. The guard against that lived in one function while another path bypassed it. The model could name a belief class its own wire evidence forbade, then invent a parameter (wait_s, the knob turned, not the thing mishandled). Replay double-counted, charging two beliefs for the same throttled call: 219 attributions over 115 real calls.

Each is now pinned by a test that names the incident it prevents.

The hardest moment was deleting our own best result. Precision reached 1.00 over five confirmed beliefs. Then the adversary dismantled all four standing at the time, and a separate audit showed three had been settled with an instrument that returned the same reading regardless of what the world did. Both were right. Six probe records moved to probes/invalidated/ and the beliefs went back to being hypotheses. A project about documentation that lies does not get to keep a number it knows is unsupported.

Accomplishments that we're proud of

The adversary rejected 4 of 4 confirmed beliefs, and was right every time. On "the rate limit is global, shared across all users": all ten calls came from one client with one credential, so global was never varied, and per-key, per-user and per-IP limits produce identical single-client evidence. On "the item is created in a draft state": zero rows before and after a single 0.5s wait never observed a draft status and never confirmed the create had succeeded. Each objection arrived with a concrete attack, and the probe designer now prefers the adversary's experiment to inventing a weaker one.

This is the thesis turned on itself. The proposer asserted causes it had not tested, which is the exact failure the system exists to catch in documentation.

And then the adversary did it too. While correctly rejecting a truncation belief for citing evidence that contained no title-length data, it described that evidence as a header_burst run. jq -r .template probes/probe-000.json returns timing. It asserted a detail it had not verified, in the middle of catching someone else asserting a detail they had not verified. We left that in the repo rather than quietly correcting the write-up.

The published spec labels its own contested entries. Two carry a warning naming the objection against them. They stay published, because withholding them would be its own kind of lie, but a reader can see which claims stand on evidence a reviewer rejected.

We reported a null result honestly. The memory-wipe ablation, including a token-length-matched shuffled arm to separate "the knowledge helped" from "a longer prompt helped", came out 1/3 across all three arms at n=1. The tool prints its own reading: this run gives no evidence that belief memory is driving performance.

What we learned

Predicting before observing changes what a model can get away with. Almost every bad verdict we saw came from a model reasoning backwards from a result.

An adversary weaker than the proposer is worse than no adversary, because it shares the blind spots and rubber-stamps. The asymmetry was the whole point. And as the header_burst slip shows, the strong adversary needs its own claims checked too, which is an argument for a third pass rather than for trusting the second.

An experiment needs a wall-clock budget as much as a call budget. Ours had neither at one stage: a design asked for a 2700-second pause and the probe did exactly as told, sleeping for forty-five minutes and blocking a worker.

What's next for skeptic

The Notion adapter imports cleanly against the real API surface with writes disabled by default, but was never pointed at a live workspace, so nothing about Notion is claimed. Running the same loop against an API we did not author is the real test of generalisation.

Beyond that: cross-tool transfer, where a belief abstracted from one tool's failure preempts the analogous failure in another, and drift canaries that re-probe confirmed beliefs on a cadence so the published spec stays true after the demo ends.

Built With

• agent-orchestrator
• claude-code
• codex
• fastapi
• glm-4.7-flash
• httpx
• javascript
• model-context-protocol
• notion-api
• pytest
• python
• tensormux
• uvicorn