Trace

Inspiration

When an AI agent says it finished a task, its own log often doesn't show what really happened. A tool call can time out, a policy can block a write, and the agent can still report success. Engineers then spend hours lining up the agent's account against the runtime records by hand. We wanted a tool that does that lining-up and shows its evidence.

What it does

Paste an agent's run trace, the expected behavior, and optionally the runtime records (for example OpenShell audit lines). Traceback sends them to Nemotron 3.5 Lightning on Nebius Token Factory and gets back:

• A plain-language summary and findings, each rated by severity.
• Evidence for each finding: the exact trace or runtime line, plus a short quote.
• What is uncertain, what evidence is missing, a recommended fix, and a replay check to confirm the fix.

Traceback checks every citation against the line it points to. Citations that don't match are dropped, and a finding with no valid citation is marked informational. It also scrubs keys, bearer tokens, and email addresses from its output.

Traceback is read-only. It never changes a policy, stops an agent, or runs a suggested fix.

A second mode, Patrol, watches a simulated fleet of six agents in real time. A deterministic radar checks each event against the agent's limits, and officer agents on Nemotron investigate each hit and write cited tickets. In the escape simulation, Traceback's tripwire triggers the runtime's emergency stop when traffic leaves a sandbox. The fleet and runtime are synthetic.

How we built it

• Analysis: one server-side request per investigation to the Token Factory chat completions endpoint, in JSON mode, with one retry on a format error. The API key stays on the server.
• Citation checking: validateCitations checks each cited source, line number, and quote against the submitted text. This checks consistency with the pasted text, not that the text is authentic.
• Patrol: a deterministic radar rules engine runs on every event. An officer agent loop (up to five steps: read lines, check policy, write ticket) investigates each hit, and every ticket citation is verified.
• Demo video: a 57-second walkthrough recorded against the live API. Screen captures of the real app are composited into a designed storyboard with FFmpeg.
• No dependencies: the server, UI, and test scripts use only the Node.js standard library.

Challenges we ran into

• Reasoning text in responses. Nemotron can write its reasoning into the answer and run out of tokens before the JSON. We turn off the thinking mode for these calls and retry once if the JSON is malformed.
• Live patrols cost credit. Each live patrol makes several model calls per investigation, so the server allows one live shift at a time.
• Runtime sources without a schema. NVIDIA's public material describes Sentry's record types but not a schema or API. We treat pasted Sentry text as illustrative and label it that way in the UI and the demo.

Accomplishments that we're proud of

• A live Token Factory run passes an automated test (npm run eval:live), and the result is saved in the repo.
• Every finding in the demo is traceable to a line the user can check.
• The local smoke test runs with no API key and makes no external calls.
• In the patrol simulation, the emergency stop fires within milliseconds of the tripwire, and the run records each action it prevented.

What we learned

A trustworthy incident report has to show its evidence. Asking a model for citations is not enough; the citations have to be checked against the source before anyone reads them. We also learned that a model's own account of what it did is weak evidence, and that runtime records are much stronger.

What's next

• Route the final analysis to Nemotron 3 Ultra and fast checks to Nano or Super, as the track brief suggests.
• Deploy to Nebius Serverless Endpoints, and run the patrol as a Serverless Job.
• Add a live adapter for OpenShell audit output, once its format is published or documented.
• Test on real agent traces from users, not only synthetic incidents.

Built With

• chrome
• css
• devtools
• factory
• ffmpeg
• html
• javascript
• nebius
• nemotron
• node.js
• nvidia
• openshell
• protocol
• token