Inspiration
Palm reading is everywhere in India. An elder at a family function takes your hand, or a reader outside a temple does. Most of us have had our palm read at some point. Very few of us know what the old texts actually say, or which parts of it are old at all.
Hast Rekha (हस्त रेखा) is the Indian tradition of reading the hand, a branch of Samudrika Shastra, the old study of the body's marks. We wanted a way to learn it on the one surface it is about: your own hand. A book or a phone app can show you a drawing of a palm. A Quest 3 with hand tracking and passthrough can draw on yours.
When I tried it on our headset tonight, Panditji looked at my right hand and called it an Air hand (Vayu Hast). He read my short head line as a mind that decides quickly and directly, my head and life lines starting together as someone who looks before they leap, and my fate line rising straight from the wrist as a strong sense of direction. Then he reminded me, as he reminds everyone, that this is culture and story, not a forecast. I'll let my teammates decide how close he got.
What it does
A reading takes about five minutes, seated, with your hands only.
- The opening. The nine grahas (the Navagraha) circle around you while a calm voice, Aakashvani, says that their names are written on the hills of your palm. Then Panditji, an original scholar character, greets you.
- Your hand is measured. Hold up your palm and the app reads its shape from hand tracking: palm squareness, finger length against the palm, index against ring finger, how the thumb sits. From that it names your hand's element, as the tradition does.
- The lines appear on your skin. Lines and mounts are drawn in light on your actual palm and follow it as you turn your hand.
- The mounts are planets. Touch a mount with a fingertip of your other hand and the room opens into that planet's sky. Panditji tells you what the old books say about it from inside that world.
- You read your own lines. For each line, cards show 2 to 4 shapes on a natural palm. You look at your own hand and pick the closest one, and the candidate glows on your real palm while you compare.
- Every line is a world. Touch or pinch a line and a portal opens out of your palm: the heart line becomes a lotus pond, the life line a banyan by a river, the head line a sky of stars joining up, the fate line Shani's ringed planet above a road of light. Close your fist and it folds back into your hand.
- Something to keep. You get a Rekha card with your element and line shapes, saved only on the headset.
Everything is voiced and captioned in English, with a full Hindi mode.
We are careful about one thing: the app does not read your creases with a camera. It measures the shape of your hand from the tracked joints, and you confirm your line shapes by comparing them with your own palm. Panditji says so during the reading.
Why mixed reality and hand tracking matter here
- Your palm is the canvas. The overlay is built in palm coordinates from four tracked joints (wrist, middle knuckle, index and pinky knuckles) and rebuilt every frame, so it stretches and turns with your hand. On a flat screen, or in VR without passthrough, there is no palm to read.
- Your hands are the controller. There are three gestures: show your palm, touch or pinch a line, close your fist. No controllers.
- The worlds keep your hands real. When a world opens around you, your hands are masked back into passthrough, so you still see your own palm inside it with the lines on top.
How we built it
- WebXR on Quest Browser, using Meta's IWSDK with Three.js and TypeScript. It runs from a URL with no install.
- A cited rulebook. 63 hand features and 212 rules from Cheiro (1916), Benham (1900), Varahamihira's Brihat Samhita (chapter 68) and modern Indian guides, each with its source. Every sentence Panditji says about your hand points at exactly one rule, and a test checks that every pointer resolves.
- Safety enforced in code. A reading is dropped if it mentions death, lifespan, illness, fertility, money loss or anything phrased as a prediction ("you will"). This is checked by tests that include 4,000 fuzzed hand profiles and choices.
- Line shapes from real data. The candidates vary the average heart, head and life lines of 399 palms, produced by an offline image pipeline.
- Panditji is built from MakeHuman (MPFB) in Blender by script and animated procedurally: he breathes and blinks, looks at your palm while it is up, moves his jaw as he talks and greets you with a namaste.
- His voice is an original voice we designed with ElevenLabs Voice Design from a text description, nothing cloned. All 96 lines are pre-rendered in English and Hindi and run through a temple reverb and tanpura drone. The 14 music loops (one per graha and per world) were made with ElevenLabs Music. Nothing calls an API at runtime and nothing leaves the headset.
- Testing. A scripted, hands-only emulator run plays the whole reading end to end (56 checks). For the real headset we wrote a small debug relay that sends frame rate, hand tracking state and errors from the Quest back to our laptop over Wi-Fi. On our Quest 3 it ran at a median of 74 fps over about 8 minutes of logged play, with short dips when a world first loads that we are still tuning.
Challenges
- Reading real creases. Quest Browser does not give web pages the passthrough camera frames, so a camera crease scan was not possible in WebXR. Instead of faking detection, we made comparing your own palm with the glowing candidates part of the experience.
- A palm held up to your face is a hard pose for hand tracking. In our first headset session the overlay blinked off whenever the palm tilted to be touched, and a pinch sometimes opened the line next to the one you meant. We added hysteresis to the palm (picked up at about 55 degrees, kept until about 78) and made a pinch open the line that is lit, not the nearest one.
- Talking about a living tradition without predicting anything. The app frames everything as what the tradition says, filters out harmful claims in code, and at the end Panditji explains why readings feel personal (the Barnum effect).
What we are proud of
- Lines drawn on your real palm that stay on it as you move.
- Every statement about your hand traces back to a cited source, and the harmful ones cannot be shown.
- On the headset it measured my hand three separate times and gave the same profile each time.
What we learned
- The emulator was not enough. The real headset showed problems we could not have found otherwise, from the palm tilt to audio staying silent when you enter through the browser's own button.
- Some of what feels like ancient palmistry (the heart and head lines, the planetary mounts) reached Indian practice through Western books of the early 20th century. Panditji now says that too.
- Asking people to look at their own hand turned out to be more honest and more engaging than pretending the headset can see their creases.
What's next
Working now: hand shape measured from tracking, line shapes you confirm yourself, the mount and line worlds, the Rekha card, and English and Hindi.
Planned:
- A camera crease scan if Quest exposes camera frames to the web. Our offline pipeline already traces heart, head and life lines from photos (on 300 photos, 148 passed its quality check), but it has no human ground truth yet.
- Checking the content with people who know it. We are not scholars of Samudrika Shastra. We want to work with scholars and practising palm readers to review the readings and the framing, and with native speakers to review the Hindi, which was translated with AI.
- More hands. The measurement thresholds come from photo datasets, not from Quest joint data, so we need to try it on many more people.
Built With
- basis-universal
- blender
- elevenlabs
- ffmpeg
- hand-tracking
- iwsdk
- javascript
- ktx2
- makehuman
- mediapipe
- meta-quest-3
- mixed-reality
- mpfb
- numpy
- passthrough
- pmndrs-uikit
- python
- quest-browser
- scikit-image
- shapely
- three.js
- typescript
- vite
- web-audio-api
- webxr
