-
-
Own-room mode: memories go on your real desk and shelves
-
Type a topic; the agent plans one picture per idea, live
-
Five causes of World War 1, five vivid images
-
Smriti hides everything and tests you; ticks and crosses, not just colour
-
Palm-up menu: repeat, voice, exit
-
Smriti: a memory palace that builds itself
-
A miss: the agent redraws Nationalism so it can’t be confused with Militarism
Inspiration
I'm a Computer Science student in Bengaluru, in the middle of campus placements. Every online test and interview asks the same theory: OOP, DBMS, operating systems, networks. I revise my notes at night, and a week later most of it is gone.
Memory champions don't reread. They use the method of loci: they place vivid, strange images along a route through a familiar place, then walk it in their head. It works, but building a palace takes hours of imagination most students don't have. In 2018, University of Maryland researchers found people recalled 8.8% more with a memory palace in VR than on a desktop. So I asked: what if the palace built itself?
What it does
You type what you want to remember, or paste your notes. Smriti ("memory" in Sanskrit), a little brass lamp with a face who lives in a Mangalore-style home, picks the facts worth remembering and imagines one vivid image for each. For the causes of World War 1 that's a glowing red shield for Militarism, a floating bridge for Alliances and a giant gold planet for Imperialism.
You pinch each image and carry it to a spot in the room: the jhoola, the bookshelf, the window sill. Placing it with your own hands is the point, because the movement becomes part of the memory.
Then Smriti hides everything and walks you through the room: "What did you leave by the door?" You touch one of three floating answers. Miss one, and Smriti's agent looks at what you confused it with and redraws that picture so the two can't be mixed up, then asks again at the end.
Your palace is saved, and Smriti brings you back on a spaced schedule (1, 3, 7, 14, 30 days) with a daily streak, a calendar reminder and a link to share your palace with a friend. In passthrough you can build the palace in your own room, and memories snap onto your real desk or shelf. Nine ready palaces (OOP, the OSI layers, Fundamental Rights, the Mughals, and more) open instantly.
How I built it
- Meta Immersive Web SDK (WebXR, Three.js, an entity-component system). Everything is hands-only: pinch to grab, touch or point-and-pinch to answer, and a palm-up menu read from the hand joints every frame.
- A real agent with checked tools. Gemini plans the palace by calling
plan_palace,create_imageandfinish_palace. My server checks every call (a taken spot, an unknown picture type, finishing early) and sends back a reason, so the model fixes its own mistakes. Progress streams live as NDJSON, so the room fills while you learn the pinch. After a miss, a second tool,reimagine_image, must design a picture that separates the two answers. - Hedged model calls. An overloaded model hangs instead of failing, so the redraw starts the fast model at once and launches backups every 2.5 s. The first valid answer wins, always within 13 s.
- 20 procedural image kits (rocket, key, scales, a parrot passing its crown…), each with a colour and an effect: giant, glowing, spinning, on fire.
- Your own room: two WebXR hit-test probes find real surfaces, and a memory released within 45 cm of one lands on it.
- Performance: static geometry is merged by material, cutting the room from 185 draw calls to 39. Marigold petals are one pooled instanced mesh.
- Sound and access: Smriti reads every caption aloud. Chimes and a tanpura drone are synthesized with Web Audio. There are switches for larger captions, calm motion, voice and sound, and right/wrong answers show a tick or cross, not just colour.
- No headset needed to try it: a mouse preview runs the whole loop on any computer. Hosted on Vercel; the API key never leaves the server.
Challenges I ran into
- Making AI output safe to render. The agent can only choose things the room can actually show, and the client checks the result again.
- Hiding the wait. The first-pinch tutorial runs while the agent is still building.
- Model overload. Under load, redraws silently timed out, so I replaced sequential retries with parallel hedged calls.
- Captions. The VR caption font only has basic Latin glyphs, so text is folded to safe characters. In your own room, stories that mention "the jhoola" are trimmed to fit any room.
What I learned
The memory technique only works if you do the placing, so the AI imagines and the hands remember. An agent is only as good as the checks around it. And designing for hands means one gesture, taught first, before any content appears.
What's next
Tuning pinch distances and caption placement on Quest hardware with real hands, voice input for topics, palaces a teacher builds once for a whole class, and more rooms modelled on real Indian homes.
Built With
- ai-agents
- function-calling
- gemini
- gemini-api
- hand-tracking
- hit-test
- iwer
- meta-immersive-web-sdk
- meta-quest
- node.js
- passthrough
- three.js
- typescript
- vercel
- vite
- web-audio-api
- web-speech-api
- webxr