Black Box - Mechanistic Interpretability for agents

Inspiration

Enterprises are pulling back from giving AI agents real autonomy — not because the agents aren't capable, but because there's no way to verify a claim of success before it triggers a real consequence: a payment, a customer message, a ticket closure. Recent research backs this up starkly: most multi-agent system failures are silent — they don't trigger an error, they just quietly produce the wrong outcome. In some benchmarks, agents report task success on the majority of cases where they actually failed. We wanted to build the layer that catches that gap before it reaches production.

What it does

Black Box is a real-time observability and audit layer for enterprise AI agents. It:

• Watches every reasoning step an agent takes as it works through a task
• Flags the moments where an agent's action isn't actually backed by the evidence it retrieved — an unsupported claim, not just a low-confidence guess
• Replays the flagged decision visually, so a human can see exactly what the agent concluded and why that conclusion didn't hold up
• Corrects confirmed mistakes by writing a rule back into the agent's working context, so the same failure mode doesn't recur
• Measures whether that correction actually worked, using a ground-truth scenario harness we built ourselves — so our accuracy claims are numbers we can defend, not a demo we staged

How we built it

The core agent runs on Claude, using tool use against a small set of simulated enterprise actions (ticket lookup, order status, refund issuance, ticket closure). Every step — the agent's reasoning, the tool it called, what came back, and a confidence signal — is logged.

On top of that sits our audit layer: a detector that flags a step when the agent's action isn't supported by its own retrieved evidence. To validate it honestly, we built a small ground-truth sandbox — a seeded set of enterprise scenarios, some clean and some containing deliberate traps, each with a known-correct outcome — so we can score the agent's actual accuracy, and our detector's accuracy at catching its mistakes, batch over batch.

The UI is built around a single visual idea: a stylized brain, where each reasoning step is a neuron, and a signal travels between them as the agent thinks. When it hits an unsupported step, that neuron flares red and holds — the moment you can click into to see the actual reasoning that went wrong.

We designed the system to extend to Freshworks and Dodo Payments — the audit layer sitting in front of agents built on their platforms, gating exactly the kind of consequential action (a support resolution, a payment) that shouldn't fire on an unverified claim. We're also scoping an extension to Sarvam's agentic models, given their published focus on tool-use benchmarks — auditing an agent built on Sarvam's stack is a natural next step for the project.

Challenges we ran into

The honest one: detecting "false success" in agent behavior is an open research problem, not a solved one. Published work testing this with LLM-judge methods couldn't reliably beat a coin flip on the hardest benchmarks. That reshaped our whole approach — instead of overselling detection accuracy, we built a system that's upfront about what it catches, what it misses, and backs every claim with a number from our own test harness rather than a curated demo.

We also had to resist scope creep more than once. Reinforcement learning and live model fine-tuning both sounded like the "right" way to make an agent learn from its mistakes — but both need far more time and compute than a hackathon allows, and claiming either without being able to show the real training run would fall apart under any technical question. We scoped "learning" down to something honest and still real: a policy-level correction loop, measured against ground truth, not a training pipeline.

Accomplishments that we're proud of

• A ground-truth scenario harness that turns our accuracy claims into measured numbers, not marketing
• An audit detector that's explicit about its limits instead of overclaiming
• A visual language — the firing/flagged neuron — that makes an abstract failure mode immediately legible
• A correction loop that closes the story: catch it, explain it, fix it, prove the fix helped

What we learned

How genuinely unsolved the "silent failure" problem is in the agentic AI space right now, even at research labs with far more resources than we had this weekend — and how much more credible a system becomes when it's honest about that instead of pretending to have solved it.

What's next for Black Box

• Extending the audit layer to a Sarvam-built agent, given their models' focus on tool-use and agentic benchmarks
• Deeper integration with Freshworks' MCP Gateway and Dodo's agent-payment stack, so the audit layer can gate real actions, not just simulated ones
• A richer correction loop, including human-in-the-loop review before a correction is applied automatically

Built With

• agentic-ai
• ai-observability
• anthropic
• claude
• dodo-payments
• fastapi
• freshworks
• javascript
• llm-agents
• mcp
• python
• react
• svg
• tool-use