Oculith

The home view: every run across our agents in one place, with regression cases and reliability stats building up over time.
What a single run looks like inside: every model turn and shell command in one tree, plus exit codes and events for any span you click.
How it all fits together: thin adapters at the starter kit's seams, redaction before anything touches disk, and the verify loop on top.
Honestly our favourite screen: a run that failed, the banner telling you exactly where, and one click to land on the failing span.
A good run is too useful to throw away, so we save it as a regression case; it gets replayed in a fresh workspace and has to stay green.

Try it live

A hosted instance is running for judging: https://oculithtechjam.duckdns.org

Access token (paste once into the unlock screen): d00788353ec5c0a87876d7b6340caf32f1a7b934ace9d428

Inspiration

It started with one bad night. Our agent died mid-task and the log gave us a single word: "error". The model? A tool call? The sandbox? We spent the evening guessing. That is the Track 1 starter kit's gap by design: a Run shows you its final message or a one-line error, and nothing connects the HTTP request, the service, the runner process, Codex's event stream, and the result. The obvious fix, logging everything, is worse, because now the logs are full of secrets. The brief leaves the trace timeline and audit model out on purpose and calls them middleware. Oculith is that middleware.

What it does

Every agent Run becomes one correlated, privacy-safe trace, and everything else is computed from those stored facts. If Oculith didn't observe something, it says "no evidence". It never guesses.

Instrument. Every real seam (Fastify, AgentService, AgentRunner, the runtime container, the Codex CLI, the workspace) emits one versioned ObservationEvent contract through a single redaction boundary. Secrets are scrubbed before anything touches disk, and the redaction fails closed.
Observe. A Runs list that surfaces failures first, and a trace view (span tree, timeline, span drawer) that shows which layer broke and why: exit-code hints, spans left incomplete by a server crash, sandbox denials, the model-vs-tools time split.
Audit. Actor, action, resource, outcome, with every row linked back to the stored event it was derived from.
Verify. One click turns a good Run into a regression case: assertions pre-filled from its own evidence, replayed in a fresh workspace, with a deterministic comparison that flags PASS→FAIL as REGRESSION. Reliability charts accumulate per-agent completion, failure, and latency over time.
Judge. LLM evaluators (task completion, recovery quality, and user-defined judges; our demo adds a politeness judge) score historical Runs 1–5. The judge is deliberately fenced: it sees only the redacted view, deterministic verdicts are ground truth it cannot overwrite, and no LLM ever writes a diagnosis. LLM as a judge, never as a witness.

One correlated trace: span tree, timeline, and the span drawer with per-span evidence One Run, one trace: the span tree with the drawer open on a tool span, exit code and events included.

How we built it

Oculith architecture: starter-kit seams, the redaction gate, and the verify loop The one-page architecture: instrument the real seams, redact before storage, derive from stored facts, verify.

Architecture. We built on the starter kit's own extension seams; the UI shell, control plane, and runtime are unchanged. All new server code lives in one module (context, emitter, redaction, NDJSON trace store, query/rollup), fed by thin adapters at the existing seams. Every event shares a versioned envelope with stable IDs and a sequence number, so one Run's activity across five layers correlates into a single tree. The emitter is non-blocking: if the trace store fails, the Run is marked degraded and completes anyway.

Observability must never break the thing it observes.

Storage and queries. One append-only NDJSON file per Run plus an in-memory index, with an optional PostgreSQL backend behind the same store interfaces. Writes are atomic and serialized, and everything the UI shows is a pure function over the stored events, so any number on screen can be recomputed from the trace file.

Frontend and quality. The React app gains the Runs overview, trace detail with span drawer, reliability charts (hand-drawn inline SVG), and evaluation views, updated live over authenticated SSE with a polling fallback. Strict TypeScript end to end, one zod schema at the boundary, 600+ automated tests against a fake runner behind the real interface, and a Playwright E2E lane on the actual Docker image with a seeded-canary privacy sweep. CI runs the full check on every push.

Challenges we ran into

An undocumented event stream. Nothing specifies what codex exec --json emits, so we captured raw streams and checked them in as fixtures before writing any mapper. That caught a real error: an expected file_change event never appears, so workspace changes are detected from the disk itself.
A deterministic regression demo. Rigging instructions to break the agent failed three times: the model fixed the bug regardless. So we redesigned the task: its tests assert SHA-256 checksums whose key business fact exists only in the agent's instructions. Removing that one line makes any agent fail, every time.
Privacy without losing usefulness. Raw capture does not exist as a mode, and redaction was verified adversarially: canary credentials seeded into commands, outputs, and model messages are never served back from any API, file, log, or the DOM.
Correctness at the edges. A Run interrupted by a server crash must persist as cancelled with its incomplete spans, neither disappearing nor appearing finished. That took atomic writes, an append-only format that tolerates termination at any byte, and startup reconciliation of late events.
Deterministic failure demos. Judges should not wait on a live model to see failure paths, so the fixtures (timeout, exit codes 2/126/127/130/137) run through the real Run path but are gated off outside the demo.
Windows and containers. No Landlock under Docker Desktop, Git Bash rewriting mount paths, and cleanup needing SIGTERM-then-SIGKILL escalation. The scripts detect each case and the docs state the fallbacks.

Accomplishments that we're proud of

Failures read like diagnoses, not log dumps. First failing step, the layer it happened in, an exit-code hint, and a Jump-to-failing-span button, with every claim linked to its stored event. A layer that exposed nothing says "no evidence" instead of a guess.
The regression story works end to end, deterministically. The knowledge-gated demo task turns a one-line config change into a guaranteed test failure, so save-case, rerun, and compare produce a genuine REGRESSION on demand. Rehearsed twice on the judged Docker path from a clean clone: 171 and 168 seconds.
The privacy sweep survived adversarial review. Canary credentials were seeded into commands, outputs, and model messages, and zero leaks were found across every API response, NDJSON file, export, server log, and the rendered DOM. Raw capture is not a mode we turned off; it is a mode that does not exist.
The verification record is real. 600+ automated tests green in CI, a 180-check E2E lane against the production Docker image, and ten recorded UAT rounds with the evidence attached to each.
The judged path is one Docker command. No cloud setup, no hidden state, no dependency beyond the Ark API key.

What we learned

Observability for agents is a trust problem before it's a UI problem. The first time a dashboard guesses ("the model probably failed"), people stop believing everything else on it. The fix isn't better wording; it's keeping observed facts, derived diagnoses, and evaluator opinions mechanically separate in the schema itself, so the product can't blur them.

Capture fixtures, never guess. The file_change event we almost invented taught us to treat every external interface as something you observe and pin down with checked-in fixtures, not something you assume. The same habit shaped the tests: real services on temp directories, a fake runner behind the real interface, and an E2E lane on the actual Docker image, because mocks can't catch the bugs that live at the seams.

Types and schemas are cheap insurance under deadline pressure. Strict TypeScript plus one zod schema at the boundary meant a whole class of 3am bugs never happened. When every event is validated against a versioned contract at the door, the query layer downstream gets to be boring, and boring is what you want in the part that computes the numbers judges read.

Determinism is a design skill, not a testing trick. The knowledge-gated demo template, the gated timeout fixture, the deterministic fake judge for CI: each exists because "run it and hope" is not a demo plan. Making a system provably show its failure paths on command turned out to be some of the most interesting engineering in the project.

We gained a deeper understanding of how AI Agents work. We gained hands-on experience building AI agents and a deeper understanding of how agentic systems reason, use tools, and execute tasks.

We learnt about the infrastructure required to make agentic systems reliable in practice. We learned about observability, tracing, middleware, telemetry, and debugging nondeterministic systems.

What's next for Oculith

Control, built on top of evidence:

Policy checks that gate Runs on trace-derived facts.
An AI agent that evaluates the metrics, logs, and traces themselves and proposes root-cause diagnoses, always as annotations linked back to spans, never as facts.
Richer evaluator packs, OTLP export into existing observability stacks, and multi-instance trace aggregation.