What it does

Rookwise investigates an operational incident for small teams that have no dedicated SRE. You give it the logs, metrics, a deploy diff, a runbook and a ticket, and you set a hard budget in dollars. It returns a diagnosis in which every claim is tied to a quoted log line, a list of the evidence it never read, the exact cost, and a recommendation that a human must approve. Version 1 executes nothing and holds no credentials.

Why it matters

A team of five cannot absorb a wrong diagnosis at 3 a.m.: there is no rotation to catch it and no budget to rerun it. AI agents can read twenty thousand log lines, but they can also cite lines that do not exist, and their bill arrives after the run. Rookwise is built around poka-yoke, the mistake-proofing idea formalized by Shigeo Shingo in the Toyota Production System: do not ask people to be careful, make the mistake hard to commit and easy to see.

How it works

A director plans the investigation. Cheap operators each read one slice of the evidence. Managers consolidate. The director writes the diagnosis. Plain code checks every citation: the source exists, the lines exist, the quote is in them. A claim that fails is withdrawn and shown with the reason. A separate auditor reviews what remains. A person approves.

A ledger reserves money before every model call and refuses any call that would break the cap.

How we built it with Nemotron and Token Factory

Everything runs on NVIDIA Nemotron through Nebius Token Factory, one OpenAI-compatible endpoint: Nemotron-3-Ultra as director and auditor, Nemotron-3-Super as managers and escalation, Nemotron-3.5-Lightning as operators. Because one endpoint serves several sizes and prices every token, a run can mix a large judge with many small readers, and the ledger can reserve money in advance. On the test split the real cost was about a quarter below the prediction. The demo site replays a recorded real run in the browser: no sign-in, no model calls.

Results, and what they do and do not show

Six synthetic incidents with known root causes, three development and three test (not used while tuning), each run once in three configurations under the same 0.50 USD cap: the hierarchy, a single Nemotron-3-Ultra, and a single Nemotron-3.5-Lightning.

On the test split the hierarchy was correct on 2 of 3 incidents, the single Ultra on 0 of 3 and the single Lightning on 1 of 3. On the development split, 3 of 3, 1 of 3 and 1 of 3.

This is three incidents per split and one run per cell. It is a signal, not statistics, and it does not prove the hierarchy beats a cheap single model. The single Ultra mostly lost by reading too little under an equal cap. A run with six verified claims still named the wrong category, so verified citations are not a proof of correctness. The corpus is synthetic.

Challenges

Models cited lines that do not exist, truncated their replies and ignored structured output rules. The harness retries, repairs invalid output with the exact validation error, escalates to a stronger model and salvages truncated replies. We also had to keep results honest: we froze the engine before the test split ran and documented every change after.

What we learned

Deterministic verification catches what the model cannot see in itself. Equal money is a useful way to compare designs. And “I could not conclude” is a better answer than a confident wrong one. Our Token Factory and Nemotron findings, each backed by the calls that show it, are in FEEDBACK.md.

What is next

Real incidents from real teams, more repeats per cell, and a read-only connector to log stores. Execution of fixes stays out until a team asks for it and a human gate exists for it.