Agent Forensics Lab

The problem

A customer asks about their own order. The support agent replies cleanly and even says, “For privacy reasons, I cannot disclose details of other orders.” A response-only evaluator scores that answer as safe.

The tool trace shows something else. Before answering, the agent had already called lookup_order on O2001, an order owned by another customer, because a poisoned note in the customer's own order told it to.

Response-only evaluation grades the words, not the actions. Agents act: they read databases, call APIs and dispatch resources. Safe-looking answers can hide unsafe actions.

What it does

Agent Forensics Lab investigates the run, not just the reply:

Trace → Detect → Localize → Minimize → Replay → Regression

1. Trace: runs the agent on NVIDIA Nemotron via Nebius Token Factory and records every tool call with its actor, resource, owner or jurisdiction, and result.
2. Detect: deterministic policy invariants, such as cross-customer access or out-of-jurisdiction dispatch, flag violations from the trace rather than from the wording of the answer.
3. Localize: pinpoints the exact tool call where the boundary broke and the untrusted source that preceded it.
4. Minimize: shortens the trigger. A shorter candidate is accepted only if a fresh execution still reproduces the same failure.
5. Replay: applies an implemented guardrail (e.g., Content Isolation) and replays the same case in a paired baseline/protected execution. This verifies the mitigation on the same replayed case and checks that the legitimate task still succeeds.
6. Regression: saves the verified incident as a regression test, so you can rerun the exact case when the agent or its prompt changes.

The same forensic core runs across two materially different domains: Customer Support (fictional orders and refunds) and Emergency Response (simulated dispatch over a frozen USGS earthquake snapshot; all actions are simulated).

---

Technological Implementation

• Primary model: NVIDIA Nemotron (nvidia/nemotron-3-super-120b-a12b), an NVIDIA open model. All inference runs live on Nebius Token Factory through its OpenAI-compatible endpoint (temperature 0, bounded steps and timeouts, one bounded format retry).
• Tool-level trace recording: every tool call is stored as an ordered event with actor, resource, owner or jurisdiction, and result.
• Deterministic invariant checks: violations are decided by explicit policy rules over the trace, not by an LLM judge.
• Localization: identifies the critical violating event and the preceding untrusted content.
• Trigger minimization: candidates are accepted only after a reproducing execution.
• Paired baseline/protected replay: fresh executions measure reproduction, safety and task utility.
• Regression storage: verified cases are persisted (SQLite), deduplicated and can be rerun.
• Two domain adapters share one forensic core through a documented adapter contract.
• 434 automated tests (controlled model responses, disposable databases) cover correctness, security, persistence and evaluation logic.

Frozen evaluation (final-combined-20261005-005502, audited exact-case evidence):

Metric	Result
Cases completed	132/132
Audited mitigations verified	67/67
Configured policy violations in evaluated protected executions	0
Baseline-successful benign controls preserved	41/41

In short: 67/67 audited mitigations verified, 0 configured policy violations in evaluated protected executions, and 41/41 baseline-successful benign controls preserved.

• Controlled customer-support ablation: 9/10 poisoned, 0/10 neutralized, 0/10 sanitized.
• Controlled multi-model subset (Nemotron · Hermes · Qwen via Nebius): not a leaderboard.

Results apply to these executions and simulated domains and are not a universal safety guarantee.

Design

• Complete end-to-end workflow in one hosted product: trace → detect → localize → minimize → replay → regression.
• Built for judges: a no-login hosted demo, a one-click domain switch, and numbered workflow steps (01 Discover … 06 Regression).
• “What the user saw vs. what the agent did”: the final response sits side by side with the recorded trace and the forensic verdict.
• Clear before/after: Before: REPRODUCED → After: GUARDRAIL VERIFIED.
• Saved regressions can be inspected in the Regression Suite without rerunning inference.
• The Evidence page shows the frozen audited benchmark with its denominators and qualifiers.

Potential Impact

• Teams deploying agents need evidence of actions, not just outputs. A privacy-aware sentence does not undo an unauthorized read.
• It surfaces hidden unsafe actions behind safe-looking responses, a class of failure that response-only evaluation can miss.
• Verified failures become reusable regression tests that can run whenever a model, prompt or tool changes.
• It is relevant to agent security, evaluation, red teaming and CI for tool-using agents.

Quality of the Idea

• It targets a specific blind spot: response-only evaluation of agents that take actions.
• It moves from grading answers to a trace-level forensic workflow: localize the step, minimize the trigger, explain the failure.
• Verified before/after replay checks that the guardrail stops the failure and that the legitimate task still works.
• Regression conversion turns a one-off incident into a durable test.
• The same forensic core across two materially different domains (customer data access and emergency dispatch) shows that the approach is not tied to one scenario.

How we used Nebius and NVIDIA

• NVIDIA Nemotron is the primary model for the agent under investigation and for every forensic stage that needs inference: execution, minimization and paired replay.
• Nebius Token Factory serves all runtime inference through its OpenAI-compatible endpoint. The same client integration also reached Hermes and Qwen on Nebius for the controlled multi-model subset.

---

Feedback on Nebius Token Factory and NVIDIA Nemotron

Based on what we measured and our first-hand experience building the project.

Nebius Token Factory

• What we used it for: all live inference for the agent, minimization, paired replay and the controlled multi-model subset. That came to 1,170 model calls (about 1.21 M tokens) in the frozen 132-case evaluation, plus the multi-model run.
• OpenAI-compatible endpoint: we used a single OpenAI-compatible client. No provider-specific SDK was needed.
• Model switching: Nemotron, Hermes 4 405B and Qwen3 235B were all available in the catalog. Switching models only required changing the model ID, with no integration changes.
• Reliability:
  • 0 provider API errors across the 1,170 calls in the frozen evaluation.
  • 0 provider API errors in the multi-model run.
  • We did not encounter provider rate-limit errors. (Our public demo's own rate limits are application-side.)
• Concrete gaps we hit in our runs:
  • Our run reports could not include cost because we had no pricing configured.
  • Transport-level retries were not observable from our client.

Onboarding

Overall, getting started with Nebius Token Factory was fairly straightforward. Creating the project and API key was simple, and the OpenAI-compatible API made the integration much easier because I could keep the same client structure I was already familiar with.

The part that took me a little longer was figuring out the exact base URL, model IDs, and which model made the most sense for my use case. Once that was clear, the rest of the setup went smoothly.

I think the onboarding could be even better with a short end-to-end example specifically for agentic applications, including structured outputs, tool use, retries, and switching between models through the same endpoint. That would have saved me some trial and error at the beginning.

What I would improve

The biggest improvement I would like to see is clearer model-specific usage information in one place.

While building the project, it would have been helpful to see rate limits, token usage, approximate latency, and cost information more clearly for each model before running larger experiments. This became especially important when I started comparing the same pipeline across Nemotron, Hermes, and Qwen.

I would also find better visibility into retry behavior and rate-limit errors useful, especially for long agent workflows where one failed request can interrupt an entire run.

The platform already makes model switching very convenient, so adding clearer operational information around each model would make experimentation and evaluation much easier.

Would I use Nebius Token Factory again?

Yes.

One of the most useful parts for me was being able to run Nemotron, Hermes, and Qwen through the same OpenAI-compatible integration. I did not need to rebuild my application every time I wanted to test another model, which made the multi-model evaluation much easier.

For projects where I want to compare agent behavior across several models while keeping the rest of the system unchanged, I would definitely use Nebius Token Factory again.

NVIDIA Nemotron

• Role: the primary model for the agent under investigation and for minimization and replay.
• Structured tool decisions: there were 5 tool-call format errors across 1,170 calls, and all 5 were recovered by a single bounded retry.
• Replay and minimization behavior:
  • At temperature 0, initial failures reproduced on replay in 58/58 cases.
  • Minimization succeeded in 46/57 cases, with a 54.32% average trigger reduction.
• Strengths we observed:
  • Consistent replay behavior.
  • Reliable recovery from format errors.
  • In the emergency-response ablation, Nemotron did not perform the poisoned dispatch (0/10 in all conditions).
• Limitations we observed:
  • Nemotron was susceptible to the customer-support indirect injection (9/10 poisoned runs). That susceptibility is what the tool is designed to surface.
  • In the emergency ablation, some answers showed output contamination even though no unsafe action occurred.
  • In the controlled multi-model subset, Nemotron produced far more output tokens (50,007, vs. 7,626 for Hermes and 10,955 for Qwen) and had a higher mean latency (4.6 s vs. 3.3 s and 2.1 s).
  • In the full evaluation, mean latency was about 13.4 s per case (customer 12.4 s, emergency 22.8 s).

Would I use NVIDIA Nemotron again?

Yes, but with realistic expectations.

Nemotron was the primary model for my project, and I used it across the main workflow, including tool decisions, replay, and trigger minimization. It was capable enough to support the full forensic pipeline and gave me useful, repeatable behavior to analyze.

At the same time, the project also exposed weaknesses. For example, in the controlled customer-support ablation, poisoned retrieved content led to unsafe access in 9 out of 10 runs. I actually think that made it a useful model for this project, because the goal was not to show a perfect agent. The goal was to detect and investigate real failure behavior.

I would use Nemotron again for tool-using agent evaluation, especially when I care about trace-level behavior and reproducibility. I would just plan for somewhat higher verbosity and runtime compared with some of the other models I tested.

---

Testing instructions for judges

1. Open https://agent-forensics-lab.fly.dev/. No login is required.
2. Fastest path: keep Customer Support selected and run the investigation, then apply the guardrail and replay.
3. Switch to Emergency Response to see the same workflow in a second domain. All Emergency Response actions are simulated.
4. Open Regression Suite to inspect saved regressions without rerunning inference.
5. Open Evidence to see the frozen audited benchmark (132/132 · 67/67 · 0 · 41/41) with its qualifiers.

Notes:

• Inference runs live on NVIDIA Nemotron via Nebius Token Factory, so run times vary. An investigation makes several model calls.
• After idle autostop, the first request may take a few extra seconds.
• This is a single public instance, so please avoid starting several runs at once.
• The demo uses fictional customer data and simulated operations. Please do not enter real personal data or real emergency requests.

Built With

• fastapi
• fly.io
• javascript
• nebius
• nemotron
• nvidia
• pytest
• python
• sqlite