AgentTrail

Inspiration

AI Agents that write files, run shell commands, and call APIs are becoming standard — but most Agent platforms give operators no way to answer basic questions after something goes wrong: who told the Agent to do that, what did it actually try, and did anything stop it? We picked the Glass Box (trace and audit) track because "the Agent did something" isn't good enough for anything beyond a toy demo — you need to be able to reconstruct exactly what happened.

What it does

AgentTrail turns every Agent Run into a correlated, attributed, cost-tracked trace instead of an opaque chat log:

Every Run is stamped with who triggered it (a self-reported human/agent identity) and which session it belongs to
Every step Codex takes — model turns, tool calls, reasoning — is captured as a span in a real parent/child tree, not a flat log
A command policy layer watches every tool call for outbound network access, credential-file access, and destructive filesystem operations; a match kills the process immediately and leaves a policy_decision span as evidence
A transient failure gets one automatic retry, visible in the trace as a recovery, not silently hidden
Secrets (API keys, tokens) are redacted from everything that gets persisted or displayed, before it's ever written to disk
Token usage becomes real cost estimates, with a per-Agent budget that hard-stops new Runs (HTTP 402) once exceeded
A Run that costs or takes 3x longer than an Agent's own historical average gets auto-flagged as an anomaly
Every edit to an Agent is versioned, and the trace shows which version produced which Run — with a diff against the previous version
One click gets an LLM-generated plain-English summary of what a trace actually shows

How we built it

On top of the provided Fastify + React + Codex CLI starter kit. The core insight was treating Codex's own --json event stream as the source of truth: instead of discarding it after the final message, we parse every item.started/item.completed event into a RunSpan, correlate it by a stable trace ID, and layer identity, policy, cost, and versioning on top as the Run executes — not as an afterthought UI feature.

Challenges we ran into

The hardest constraint was that Codex executes tool calls inside its own subprocess — the platform can only observe the event stream, not intercept execution before it happens. That meant the policy layer had to be honest about being a detection-and-containment boundary (kill the process the moment a violation is observed) rather than claiming to prevent execution outright, which we document explicitly rather than overselling.

We also caught a real bug mid-build: our first pass at redacting secrets from Agent output would have fed Codex a redacted, broken prompt — redaction has to apply to what's displayed, never to what's actually sent to the Runtime.

Accomplishments that we're proud of

A live, working denial case — you can watch a policy violation happen, kill the Agent's process, and produce a piece of trace evidence, on demand, not just in a screenshot. And a real budget enforcement path that actually blocks new spend with a proper HTTP status code, not just a UI warning.

What we learned

That "observability" and "audit" are different problems that only become powerful together — a trace tells you what happened; identity, policy, and versioning tell you who and why and under what rules. Building them as one connected pipeline, rather than bolting audit on top of a trace viewer afterward, made every later feature (anomaly detection, diffing, explain) cheap to add because they could all just read the same trace.

What's next for AgentTrail

Hardening the command policy against basic obfuscation (it's currently pattern-based, not a full parser), and extending identity from self-reported to actually verified.