-
-
Palace: one phone walk becomes a 3D memory Claude can reason over.
-
Same question, real answers: Claude with one photo guesses; Claude with Palace knows where the kettle is.
-
Space as people think of it: objects, relations (on, in, next to) and places, drawn inside the 3D room.
-
"Where did I leave the kettle?" The place, when it was last seen, and the photo that proves it.
-
Every object is kept as evidence, with close-ups from every angle the walk caught it.
-
The 3D memory of a 2.5-minute walkthrough: 110 objects and 250,000 points.
-
One live memory on SpacetimeDB: Claude renames the piano and a teammate's screen updates instantly.
-
Ask out loud: Cartesia transcribes the question and speaks Palace's answer.
-
Our own phone scan of a Cal Hacks room (Gate 4), made with permission. Claude worked out it's a hackathon.
Inspiration
Claude can already read your Google Drive, your GitHub, your inbox. But it has never seen your kitchen. That's why we can't just put an LLM in a robot's head and say "bring me my keys." To act in our world, AI first has to see it, remember it and understand it.
Palace changes that. A memory palace is the ancient technique of remembering things by their place in a building. Palace builds that building for AI, from the real one.
It is meant to help people too: after a family member walks the home once, someone who is blind or has low vision could ask out loud where their glasses were last seen.
What it does
Walk through a space once with a phone. Palace rebuilds it in 3D, outlines the objects it finds, and Claude names and describes each one. A 2 min 25 s iPad walkthrough of a real home (Apple's public ARKitScenes dataset) became 110 objects, 256 relations, 31 groups and 250,000 3D points in about 2.5 minutes.
The key idea: people don't think in x, y, z. We say the kettle is on the counter, next to the toaster, in the kitchen. So Palace stores space as a hypergraph of objects, relations and places, and Claude decides which places matter. That gives a language model grounded spatial reasoning.
Ask by text or voice: "Where did I leave the kettle?"
- Claude with one living-room photo: "I can't see a kettle anywhere in this photo…"
- Claude with Palace: "…on the kitchen counter next to the stainless steel toaster…", with its last-seen time and photo proof.
A photo is one frame, one moment, and Claude's memory keeps your words, not your rooms. Palace remembers the whole space over time, with identity, relations and evidence, and admits it never saw your car keys.
What sets it apart: 3D vision, object memory, a spatial hypergraph and Claude's reasoning in one live, shared system. Efficient too: the AI asks a compact memory, not hours of video, and answers in seconds.
It matters wherever AI meets the physical world: insurers need proof of what each room held, investigators the scene as it was, stores and warehouses what moved, security teams what changed across sites.
How we built it
- See: VGGT-1B reconstructs the room from ~48 keyframes in 15–22 s. Mask R-CNN v2 and Claude propose objects; SAM 2.1 large masks fuse across views into 3D instances, which Claude Opus 5.5 labels.
- Understand: a deterministic builder derives 12 relation types plus surfaces, clusters and areas, which Claude names and rates.
- Remember: SpacetimeDB is the live product's only database: 23 private tables behind membership-gated views. Each commit is one reducer transaction applying the same pure TypeScript plans the tests run. Claude's writes are reducer calls too; every client shares live views; a commit reached another client in 20–24 ms locally.
- Answer:
claude-opus-5-5with 15 memory tools, citing evidence; any agent gets the same tools over HTTP or a local MCP server. - Speak: Cartesia push-to-talk:
ink-whispertranscribed 5.6 s of speech in 263 ms;sonic-3.6starts speaking 0.8 s after Claude's answer is ready, about 5–6 s after the question. Captions, mute, cancel, typed fallback.
Challenges we ran into
- Classic 3D fell short: COLMAP registered only 35 of this home's 116 frames; VGGT-1B poses all ~48 keyframes.
- Invented moves: two scans of an unchanged home first gave 66 ambiguous objects and invented moves. Aligning the scans (RMS 1.4% of the room) and adding position to the Hungarian matcher's SigLIP 2 appearance cue gave 51 matched, 21 ambiguous look-alikes like bar stools, 38 unmatched, 0 invented moves.
- Teaching AI to say "I don't know": a confident wrong answer hurts a blind person or a robot more than none, so Claude answers only from memory tools and decides identity only when confident.
Accomplishments that we're proud of
- Real phone video: I scanned one Cal Hacks room (Gate 4), with permission from the people in it. Asked what event this was, Claude answered from memory: "It looks like a Halloween-themed hackathon…"
- Solo, in one weekend, with Claude Code as pair programmer: ~23,000 lines of TypeScript, ~6,600 of Python, 297 TypeScript tests.
- Responsible by default: detection and labelling filter people out, though stored photos and video can still show them; Claude is told text on objects is data, never instructions; a person's chosen name beats Claude's.
What we learned
What Claude is given matters as much as which Claude: the same model guessed from one photo of one room, and gave a grounded answer from a memory of the whole home, as places and relations. And: compute what must be right; ask Claude what needs judgment.
What's next for Palace
- Testing with blind and low-vision users, not yet done.
- Robots: asked to brief a home robot, Palace gives the kettle's place and how to recognize it, but not yet the route: that needs metric scale and free space.
- Seek pilots with insurers, investigators, warehouses and security teams.
- Privacy: blurring people in stored photos; on-device processing.
If AI is going to work in our world, it first has to see it and learn it. Palace is how.
Palace's code, documentation and designs are the property of Kay Kahran Aryamehr; all rights reserved. Third-party libraries, models and data keep their own licenses.
Log in or sign up for Devpost to join the conversation.