events-playing
claimed-pov
corrected-pov
stairwell-overview
stairwell-corrected
flags
override-flags
selected-transcript
re-interview
selected-timeline

A customer tells police she saw the robber's face from the chips aisle. Rashomon places her in a measured 3D model of the store and draws her line of sight, and the ray stops at a shelf. It doesn't mark her wrong. It asks once: "From the chips aisle, the chips shelf would be between you and the suspect. Where exactly were you standing?" She remembers stepping to the front window. The event moves; the ray turns green.

Rashomon turns witness statements into a 3D reconstruction where every event links to the words it came from, and a deterministic gate, not the model, checks each one against the room. It never scores credibility, and when two witnesses disagree, it says so without picking a side. The room can be a store we modeled, or a stairwell someone scanned with a phone an hour ago.

Inspiration

Eyewitness testimony is some of the most common evidence there is and some of the hardest to check. A witness talks, the account becomes a paragraph, and nobody tests that paragraph against the room it describes. The National Research Council's 2014 report Identifying the Culprit sets out how eyewitness memory depends on viewing conditions and on what happens after the event, and that honest witnesses get distance, time and sequence wrong.

Two more findings set our limits. Chan et al. (2024) found that a generative-AI interviewer asking suggestive questions can plant false memories, so our system may not supply details or judge the witness. People v. Duenas (Cal. 2012) separates a demonstrative animation, which illustrates testimony, from a simulation that reaches conclusions, so we stay on the animation side.

We spoke with someone who has given a formal statement to the police about something they witnessed. The hardest part, they told us, was "organizing everything in chronological order, or separating things out and explaining them clearly, while also knowing whether what I said had been recorded accurately." Their biggest worry was a guess being written down as a fact: "If something I said wasn't actually a fact but was recorded as if it were, it could create confusion and have a negative impact." That is why every event in Rashomon links to the words it came from, estimated times are drawn as estimates, and the room asks a question instead of recording a conclusion. (The interview was in Korean; quotes are our translation, used with permission.)

What it does

Rashomon turns a witness statement into events anchored to landmarks in a 3D scene. Every event cites the utterances it came from. A deterministic gate checks each event against the room: can she see him from there, can he walk that way, can anyone cover that distance in that time, and does this contradict another statement. An event the room rules out is drawn as a wireframe ghost with a red badge and a neutral follow-up question. It does not become part of the reconstruction unless an investigator overrides it with a written reason.

In the demo, a customer says she saw the suspect's face from the chips aisle. The sightline ray stops at the chips shelf, and the gate asks: "From the chips aisle, the chips shelf would be between you and the suspect. Where exactly were you standing when you saw that?" She says she had stepped out to the front window. The event moves there, the ray turns green, and the replay shows her view of the counter.

Statements arrive two ways. An investigator types or pastes them into the Statement view, or a Cartesia voice agent interviews the witness, in the browser or by phone, and sends each turn to our database, which hands back the room's question for the agent to ask once. Investigators drag events to new places, and every drag is a patch that goes back through the same gate. A second witness, the clerk, adds a layer of their own. Where the two accounts disagree, the result is a soft flag naming both, not a verdict.

Any room works. Drop a GLB scan from a phone: the mesh becomes the surfaces the sightline test uses, Claude proposes landmark names and kinds from the geometry, and the investigator confirms each one before it enters the scene. We did this with a scanned stairwell. A witness said, by voice, that she watched a man by the south window from the top of the stairs; the ladder in the middle of the room blocked that ray, the agent asked where she was standing, she said she had stepped to the west window, and the flag cleared. Nothing in that room was modeled by hand.

How we built it

Contracts first. packages/schema defines landmarks, segments, events, flags, overrides and patches, plus the fixtures: the store, the robbery, the scanned stairwell and its scenario. Everyone built against the fixtures.
Engine (packages/engine). Deterministic TypeScript with no model inside.
- The resolver turns landmark anchors into positions, finds routes over portal waypoints around the shelves, and infers missing times from walking and running speeds. A scene with no portals gets a walkable graph generated from its landmarks with A* on a 0.25 m occupancy grid.
- The gate runs four checks. Sightline uses eye-to-chest and eye-to-head rays against the landmark boxes and, for a scan, the mesh itself. Time-distance uses limits of 2, 6 and 8 m/s, and is a soft flag when the time was inferred rather than stated. Reachability and cross-statement contradiction complete the set.
- The gate assigns every status, including overridden and superseded, from the data itself.
Extraction (packages/extractor). Claude Sonnet 5.5 with seven tools and a validator, tool_choice auto and max_tokens 16000. Confidence is an enum mapped to 0.3, 0.6 or 0.9 in code. A reply with no tool call, or calls the validator rejects, get one retry. A truncated reply keeps the patches already accepted. A verb outside our 22 actions maps to the nearest one and keeps the witness's own word. Each statement carries its speaker, so a clerk's "I" is the clerk, and a correction that moves a witness also completes her walk from the old place.
SpacetimeDB module (spacetimedb/server). The module stores the graph and knows each scene's landmarks.
- An editors-only submit_statement reducer queues extraction on a private scheduled table.
- The extract_segment procedure calls Claude with ctx.http.fetch.
- apply_patches, override and extraction all re-run the gate in the same transaction that writes the change, and all three are limited to editor identities.
- Three webhook routes take the voice agent's calls behind a shared-secret header. A private asked_flags table makes sure each hard flag's question is handed to the agent only once.
Voice (apps/agent). A setup script creates a Cartesia Managed Agent per scene with the webhook tools record_statement, get_scene_context and end_interview. Session and witness ids are constants the model never sees, and the scene's landmark labels are speech keyterms. The instructions follow cognitive-interview order and forbid suggesting details or judging the witness. In the browser, a voice button opens a WebSocket to the agent with a short-lived token from a small local token server, so no key reaches the page.
Scene (apps/web/src/scene). React Three Fiber in a box-model store dressed with CC0 Kenney models, or inside a scanned mesh.
- Mixamo rigs and 27 clips load through FBXLoader, with root motion stripped. Blending is computed from one timeline value, so scrubbing works in both directions.
- A procedural layer adds head look-at, a two-bone IK reach for the knife at pick-up, and walk cycles timed by distance along the path.
- Each status has its own rendering. There is a witness POV camera, and drag-to-edit emits update_event patches.
- A scan loader takes a GLB from a URL or a drop, keeps it in IndexedDB, aligns it to the floor, extracts collision geometry for the gate, and keeps textures stored in older glTF extensions. The labeling tool places and names landmarks; "Suggest landmarks" renders candidate views of the mesh, asks Claude for names and kinds, and shows each proposal as a numbered box to confirm or delete.
UI/UX (apps/web/src/ui) Built for an investigator on a laptop comparing accounts, without letting a guess look like a fact, the fear our interviewee named.
- Each visual channel has one meaning: actor color is a person, red and amber are gate statuses, white is selection, light blue is playback. Nothing is reused, so red always means the room objected.
- Estimated times get a label and a wavy boundary instead of warning colors, so an estimate reads as uncertain, not wrong.
- Flags and Events are the two review tabs. Transcript and history open from the selected event, so every event is one click from its source words. Adding a statement sits apart from review.
- We built the first screens in Figma Make, pushed them to GitHub, and had Claude Code read that code to integrate it with the repository and backend. We looped Make → push → Claude Code until the structure held, then refined details in Claude Code alone.
Process. One coordinator agent kept the board, wrote handoffs, and reviewed every branch with a fresh Opus reader (plus Sonnet for engine, schema and module changes, and Greptile while we had credits). Workers ran in cloud sessions and on laptops, overnight too. 58 pull requests merged before the freeze, each with an independent review; nobody merged their own work.

Challenges we ran into

Scrubbing backward. Mixamo clips carry root motion and real-time crossfades, and both break backward scrubbing. We strip the hips' X and Z motion at load and compute every blend weight, and now every look-at and reach, from the timeline value.
Calling Claude from the database. A model call cannot run inside a transaction. Extraction is a read transaction, then the HTTP call, then one write transaction that applies the patches and runs the gate together.
Strict schemas have a budget. Our first live extraction failed with "too many parameters with union types." Seven tools with nullable fields crossed the API's limit for strict tool schemas. We kept strict mode on the small tools and validated the large ones ourselves.
Secrets on the demo laptop. The browser cannot hold the webhook secret. Typed statements go through an editors-only reducer keyed on SpacetimeDB identities. A browser that is not an editor shows its identity with a Copy button, so it can be added to EDITOR_IDENTITIES.
Concurrent edits. Every patch batch names the version it was based on, so a stale write is refused instead of silently overwriting a newer one.
The question's wording. The gate's question had to name the obstacle, never the witness. Getting the templates right took more iterations than the raycast.
Asking once. An interviewer that repeats a challenge starts to sound like an accusation. The agent only asks the question the webhook hands it, and the module hands each one out once.
A relocation is a walk. When the witness said she had stepped to the window, the model moved her look but not always her feet, so she teleported and the gate saw her in two places at once. The fix was in code, not the prompt: a correction that moves a placed event emits the walk from the old place and shortens the earlier stand.
A scan is not a room until someone says what is in it. Our test scan turned out to be a stairwell with a ladder, not a shop. We labeled it in the tool in twenty minutes and wrote a scenario that fits the room, which is the point: the gate only needs landmarks, a blocker and a claimed line of sight.
Constraints of the voice stack. Ink-2 does not label speakers, so we pass the speaker in with every statement. Cartesia's Line SDK is being retired, and a tool call times out at 120 s, which bounds how long extraction may take. We built on Managed Agents and left two-device copilot mode for later.

Accomplishments that we're proud of

The contradiction catch works end to end, live on SpacetimeDB Maincloud, from typed text and from voice. On the frozen build, a live voice interview produced 10 statements, 0 extraction errors and 13 patches, and the spoken correction cleared the sightline flag while the witness was still on the call.
A room we never modeled works the same way. A phone scan of a stairwell, labeled in the tool, took a spoken statement, flagged a look the ladder blocks, and accepted the spoken correction.
The verdict commits with the change. No client ever renders an edited event before it has been re-checked.
The browser and the database run the same gate code, so the same graph gives the same verdict everywhere.
Every event cites its source utterances. Every extraction records the model, the prompt hash and the inputs it saw. Every override records who made it, when, and why.
Nothing in the product scores credibility.
At the freeze (tag demo-v1, commit 58d8e57), 58 pull requests had merged with independent reviews, and pnpm -r test passes 888 web tests plus the engine, extractor and module suites.

What we learned

Geometry checks are cheap and decisive. Truthfulness is not something software can check, and the product works better because it does not try.
The model maps a witness's words to landmarks reliably when the landmark list carries the aliases people actually use ("the till", "the fridge"), and a fresh room needs only its labels to work.
Rules that matter belong in code, not in the prompt. The model proposes; the resolver completes a relocation's walk; the gate decides.
Freezing the contracts and fixtures early let three people build in parallel without waiting on each other, and a coordinator that reviews every branch with a reader who did not write it caught the bugs we were too tired to see.

What's next for Rashomon

Generated motion for verbs we have no clip for (built on a branch with Uthana, held out of the demo), and body types taken from the witness's description (six rigged bodies are ready).
Scans straight from a phone into live mode: today a labeled scan is registered in the module as a scene; next is uploading the manifest from the labeling tool itself.
Door states in reachability, and recognition-distance and lighting checks as soft flags.
Copilot mode: the investigator asks, and the system listens and suggests questions.
An export that carries the override log and the label "AI-assisted illustration, not a recording."
Test it with people who have given formal statements, the community we spoke with, and then with the investigators who take them.