REPLAY

REPLAY uses WebMCP to let people and agents reconstruct road incidents together without turning uncertainty or inference into fact.

Comment

Inspiration

Most minor road incidents have no black box. The useful record is usually scattered across photographs, vehicle damage, final positions, approximate timing, and conflicting memories.

Traditional forms flatten those relationships into prose. A general-purpose AI conversation can create a different problem: an assumption or inference may gradually sound like an established fact.

I built REPLAY to explore a safer form of human-agent collaboration—one where evidence, memory, uncertainty, dispute, and agent inference remain visibly distinct throughout the investigation.

What it does

REPLAY is a local-first visual workspace for reconstructing minor, no-injury road incidents.

People can:

  • Position and rotate vehicles on a calibrated road scene
  • Create and edit timed trajectories
  • Record impacts and vehicle damage
  • Add observations with explicit provenance and certainty
  • Attach and connect local evidence
  • Track unresolved questions
  • Compare alternative hypotheses
  • Run deterministic geometry, motion, and consistency checks
  • Produce JSON, SVG, PNG, and citation-bound PDF exports

The project includes four repeatable synthetic scenarios, including a calibrated roundabout and a straight-road case with a path-derived 65–80 km/h approach.

REPLAY is intentionally not a forensic simulator, truth detector, or fault-assignment system. Its calculations are transparent review aids based on the recorded inputs.

Why WebMCP

A screenshot may show two vehicles, but it cannot reliably tell an agent which timed point belongs to which hypothesis, whether a statement is photo-backed or disputed, which geometry is locked, or whether the human has changed the case since the agent last read it.

WebMCP exposes those semantics directly from the live page.

REPLAY provides a lifecycle-aware inventory of 18–19 structured Site Tools. An agent can read the current case, inspect evidence relationships, run deterministic validation, focus an issue in the visible interface, add attributed observations, compare hypotheses, create a reversible geometry proposal, and prepare a report preview.

The agent and human do not work on separate copies. Both use the same validated domain command layer and the same visible case.

The authorization boundary is deliberately asymmetric:

Agent: read, validate, propose, and draft. Human: attest, decide, confirm, and finalize.

An agent proposal remains a visible preview until a person reviews it. No Site Tool can confirm a claim, attest that evidence or uncertainty was reviewed, accept its own proposal, decide fault, or finalize the report.

How I built it

REPLAY is a static React 19 and strict TypeScript application built with Vite.

The domain layer uses Zod schemas and a command engine for validation, optimistic versioning, provenance, locks, idempotency, hypotheses, proposals, consistency checks, undo/redo, reporting, and import/export.

The interface includes:

  • An accessible SVG scene editor
  • A synchronized incident timeline
  • Simple and expert workspace modes
  • Evidence, facts, questions, hypotheses, reports, and activity panels
  • Human-only proposal and finalization controls
  • A guided onboarding and Site Tools walkthrough

Case data and uploaded evidence remain in the browser through IndexedDB and Dexie. The runtime requires no account, analytics service, location permission, backend, or model API.

WebMCP tools are registered through document.modelContext.registerTool(...) using fixed JSON schemas derived from the same validation rules used by the interface. Durable WebMCP writes are staged, version-checked, saved, and only then adopted by the visible application.

Challenges I faced

The hardest challenge was preserving one trustworthy state across human actions, agent calls, persistence, and visible rendering.

A tool result must not silently overwrite a newer human edit. REPLAY therefore uses case versions, compare-and-swap persistence, semantic idempotency keys, pose-target guards, cancellation handling, and compensation for post-save conflicts.

Another challenge was representing uncertainty without turning the workspace into an unreadable collection of warnings. I designed explicit states for confirmed, reported, likely, uncertain, disputed, unknown, and agent-hypothesis content, while keeping provenance and open questions inspectable.

Scene geometry also required careful boundaries. REPLAY checks calibrated vehicle footprints, contact placement, road clearance, speed, acceleration, turning, and heading consistency—but labels every result as a deterministic advisory rather than proof of what physically happened.

Finally, I had to make the human authorization boundary real rather than cosmetic. The restrictions on confirmation, completeness review, proposal acceptance, and report finalization are enforced inside the domain command layer, not only through hidden or disabled buttons.

What I learned

I learned that effective agent integration is less about exposing many actions and more about exposing precise semantics, trustworthy state, and clear authority.

WebMCP works best when tools are narrow, versioned, attributable, and connected to the interface the person is already using. Read operations should explain what changed—or explicitly state that nothing changed. Mutations should preserve provenance and be safe to retry. Consequential decisions should remain visible and intentional.

I also learned that uncertainty can be useful product data. A reconstruction does not need to pretend every question has been answered before it can help someone organize evidence, identify contradictions, and prepare a reviewable record.

Accomplishments

The final release includes a complete human workflow, native Site Tools, local persistence and recovery, accessible desktop and mobile layouts, four deterministic scenarios, structured exports, and human-gated report finalization.

The release passed 463 automated unit and integration tests plus 223 end-to-end browser tests with no failures. The deployed artifact is linked to an exact clean source commit and is verified byte-for-byte after deployment.

Most importantly, REPLAY demonstrates an agent collaboration model where AI can contribute meaningful structured work without being allowed to convert its own conclusions into accepted truth.

Built With

Share this project:

Updates