Re:collection

Inspiration

Re:Collection started at home. Shivin's grandfather lived with Alzheimer's, and his family watched the visits that meant the most to us slip away from him, sometimes by the next morning. His family would show him photos and tell him the story again, but a photo only reminds you that something happened. It can't put you back in the room.

We wanted to give people like Shivin's grandfather an environment he could actually step back into: the room, the people in it, what they said, so he could relive the moment instead of just hearing about it. Research backs this up. Reviewing recordings of real events helped people with Alzheimer's recall more of them than reviewing a written diary (the SenseCam study, Woodberry et al., 2015), and reminiscence work is widely used in dementia care (Woods et al., 2018, Cochrane review).

When we looked at what exists, most of it needed too much: VR headsets, powerful computers, or a constant connection to a big server. That isn't realistic for a family visiting a parent in memory care. So we set out to make it cheap and attainable: an iPhone the family already has captures the visit, and anyone can relive it on an ordinary iPad or laptop browser. No headset, no special hardware, and nothing for the person with Alzheimer's to learn. The heavy 3D work happens once, so viewing a memory is as simple as opening a link.

What it does

Re:Collection turns a family visit, or an old home video, into a 3D memory someone with Alzheimer's can step back into.

Capture. A family member scans the room with the Re:Collection iPhone app (LiDAR), then films the visit in the same session: video, sound, the camera's exact pose every frame, and depth 10 times a second.
Or bring an old video. Any ordinary video is rebuilt into the same kind of 3D room with Depth Anything 3, and labelled "estimated from a video, not measured" everywhere it appears.
Claude finds what's true. Claude proposes small, checkable details ("Priya brought a mango cake"), each pinned to the frame or the words that prove it.
The family decides. Approve, reword, or mark a detail "don't bring this up" (for example, someone who has since passed away). Nothing reaches the patient without a yes.
The Memory Palace. Every memory hangs as an oil painting in a calm, walkable gallery. A caregiver chooses the paintings, their order and the words on each label. Step into a painting and the room rebuilds around you in 3D, with the people replaying where they stood.
A voice guide. A slow, warm voice introduces each memory and answers questions hands-free. It waits while they search for words, and it only says what the family wrote or approved.
Family, from anywhere. Relatives open the same private link on their own phone or laptop. Their screen follows the patient's walk and steps into memories with them, and they can talk to each other.
No headset. On an iPad, the Memory Window follows the viewer's head with the front camera, so the screen behaves like a window into the room.

How we built it

iPhone app (Swift, ARKit, Metal, no third-party packages). One continuous ARKit session covers the room scan and the visit, so both share one coordinate system. A live GPU point cloud is exported at 1 cm. During recording the app writes 8 Mbps H.264 video, a pose and intrinsics per frame, and compressed depth. It also makes one keyframe per second and an on-device transcript (Apple Speech) while recording. Uploads go straight to cloud storage in resumable parts and keep going in the background.

Backend (Convex + Cloudflare R2 + pull-based workers). Convex holds captures, facts, reviews, sessions, the palace and every Claude call, so no API key ever reaches a browser. A job queue with leases and heartbeats hands work to workers:

A bake worker precomputes each room once, in 3 to 9 seconds. That took the browser's time to a ready room from 6.3 s to 0.7 s, and the front page's first download from 713 KB to 99 KB.
A GPU worker (an RTX 5060 Ti and an ASUS GX10 with 128 GB) converts videos and trains a Gaussian splat of every room, phone scans included.

Putting people back in the room. For a depth pixel $(u, v)$ with depth $d$, scaled intrinsics $(f_x, f_y, c_x, c_y)$ and that frame's camera-to-world pose $T$, the world point is

$$ \mathbf{p} = T \begin{bmatrix} (u - c_x)\, d / f_x \ -(v - c_y)\, d / f_y \ -d \ 1 \end{bmatrix} $$

The web viewer (React, TypeScript, React Three Fiber) runs this in a vertex shader for every depth frame in sync with the video, coloring each point from the video. People move and talk in the scanned room, in place. Areas the camera never saw stay empty; we never fill them in.

Old videos to 3D (Python, PyTorch). The pipeline:

Depth Anything 3 estimates poses and metric depth across frames.
A second 1008 px pass per frame recovers thin details. A joint bilateral filter transfers the multi-view shape onto it without bleeding across edges.
Mask R-CNN removes moving people from the static room.
Pixels that neighbouring frames disagree on are dropped.
Points are fused at 5 mm, up to 6 million.

Checked against an iPhone's own LiDAR and ARKit poses, the camera path was off by 1.4 to 4.2 cm and depth by 1.2 to 1.3%.

The Memory Window. Instead of rotating a camera, the screen is a fixed window and the viewer's eye moves behind it. With the eye at $\mathbf{e}$ and the screen's edges at $x_l, x_r, y_b, y_t$ in eye space, the frustum is off-axis:

$$ \text{left} = \frac{n\,(x_l - e_x)}{e_z},\quad \text{right} = \frac{n\,(x_r - e_x)}{e_z},\quad \text{bottom} = \frac{n\,(y_b - e_y)}{e_z},\quad \text{top} = \frac{n\,(y_t - e_y)}{e_z} $$

That's what makes a flat screen read as depth. Head position comes from MediaPipe face detection in the browser; no camera image leaves the device.

Claude for judgement, code for guarantees.

Facts: Claude Opus 5.5 reads up to 30 keyframes and the timestamped transcript. Code keeps a citation only if it points at a keyframe Claude was actually sent, or quotes words really in the transcript.
Conversation: Claude Sonnet 5.5. Every reply is a list of sentences, each citing the facts it uses.
Grounding check, in code. Let $A$ be the approved facts and $N$ those marked "don't bring this up". A sentence $s$ citing facts $F_s$ is kept only if

$$ F_s \neq \varnothing \;\wedge\; F_s \subseteq A \setminus N $$

or if it's a short piece of small talk with no name, date, place or relationship word the person didn't just say. Everything dropped is logged with its reason.

The AI guide.

Cartesia Sonic 3.6 speaks, streamed at 0.9× speed.
Ink-2 listens hands-free with the "Patient" turn profile, so slow speakers aren't cut off. The family's names are sent as keyterms, so "Priya" is heard as Priya.
Moss keeps a memory index searched only within the memory on screen, and remembers what the patient shares ("You told me this kitchen looks like your mother's").
Claude writes each answer from what Moss found, under the same grounding rules.

The Memory Palace is a classic gallery built entirely in code (textures, frames, furniture), with an oil-painting shader on every picture that lifts away when you step in. A live visit links family screens over WebRTC, with a voice relay through Convex when a direct call can't connect.

Challenges we ran into

Honest without being robotic. A prompt can't guarantee Claude never invents a detail for someone who can't push back, so the guarantee lives in code. The hard part was not making the guide mute: words the person just said are allowed back, so it can follow their lead.
Lining people up with the room. The room and the visit must come from one unbroken AR session, and the browser's math must match the phone's exactly. Our first captures recorded at 60 fps with about 12 Hz depth instead of 30 and 10; we locked both and record the real rates.
Old videos have no true scale, and phones store video sideways. We detect the upright orientation from what's in the frame and label every estimate. Long videos drifted when split into chunks on a 16 GB GPU, so we ran them in one pass on the GX10, which cut the path error from 68 cm to 4.2 cm.
Rebuilding the backend mid-hackathon. We moved from a single laptop server to Convex, R2 and workers on Saturday night while the website and app kept changing, and merged three people's parallel work without losing any of it.
Venue Wi-Fi blocks device-to-device traffic, and the iPad only allows the camera and motion sensors over HTTPS. So everything goes through the cloud, with a voice relay for when calls can't connect.
Designing for dementia. No flashing, large type, a guide that waits, no quizzing, and a patient view with nothing to break.

Accomplishments that we're proud of

The whole loop runs on real hardware with real data: an iPhone scan, evidence-linked facts, family approval, a walkable Memory Palace, and a voice guide that answers questions.
People replay in 3D from the phone's own depth, in the scanned room, in sync with their voices.
Rooms from ordinary old videos that measure within centimetres of real LiDAR.
A grounding check we can show a judge: a sentence the model tried to say, and the logged reason code removed it.
More than 200 automated tests across the iPhone app, backend, workers and website.

What we learned

Claude is excellent at the judgement calls: which details are memorable and checkable, when someone is upset. The safety rules belong in tested code around it.
"Offer, don't quiz" changes everything about the tone, and it fits a model that's told exactly what it may say.
Labels cost little and build trust: estimated scenes are marked, dropped sentences are logged, mock output is flagged.
Moving heavy work off the browser (bake once, read small files) made the experience feel instant on an iPad.

What's next

A pilot at one memory care facility: the same visit reviewed with Re:Collection versus a written diary, recall measured at 2 weeks and 3 months (the SenseCam protocol). We'll track details recalled per memory, on their own versus after a hint.
Accounts, consent records and encryption at rest, replacing today's private links.
More languages for the guide, and filling in partly seen objects without ever inventing unseen space.