Reality is Yours

Look through your Ray-Bans, point at anything, and say what to change; AI does the rest.

Comment

Inspiration

The Ray-Ban Meta glasses put a camera where your eyes are, but all the editing and flavor is stuck in other apps. We wanted the first-person view from Ray-Ban Meta glasses to be the canvas: look at something, point at it, say what you want, and it changes the output. No screen to open, no layers, no tools.

What it does

You point at an object with your finger and speak.

“Make it pink” recolors it, keeping the object’s shading. “Blur that” and “less” / “more” adjust it instantly. “Turn that into a cardboard box” or “remove that” regenerates the object with Meta’s Muse Image. “Make the taller one red” or “the one on the left” is understood from the camera frame itself. “Only the top half” and “a bit bigger” refine the selection by voice; “undo” and “again” work as you’d expect.

It’s designed for the first-person view from Ray-Ban Meta glasses. For this hackathon the demo uses a webcam as that camera, so the edited result appears on a laptop or projector.

How we built it

Pointing: MediaPipe hand tracking with a One Euro filter for steady fingertip tracking, and a pinch gesture to select. Selection: MediaPipe interactive segmentation for a point; Gemini finds named objects from the frame; Meta’s SAM 3.1 refines masks; MediaPipe FaceLandmarker for face parts. Selection is separate from operation, so any selection works with any edit. Voice: Meta’s Muse Voice Transcribe (about 0.7s from key release to text), with an automatic Web Speech fallback. Commands: Gemini turns speech, the frame and the pointed position into a structured command. If Gemini is slow or rate limited, it falls back to Meta’s Muse Spark, which also sees the image. Edits: recolor and blur run locally in milliseconds. Remove and replace use Meta’s Muse Image through our server (keys stay server-side), tuned to about 9 seconds, with a dimmed preview and crossfade while it renders. Glasses input: an iPhone app built on Meta’s Wearables Device Access Toolkit streams the camera through its Mock Device Kit to the editor over a WebSocket relay. We had no glasses, so the phone’s camera stands in for them. Swapping the webcam for it changes nothing downstream.

Challenges we ran into

Generative edits take seconds, so we made some edits instant and made the slow one feel intentional. Models guessing wrong: a text-only model once invented object names, and a bad mask once overwrote an earlier edit. We added the frame to parsing, made failed lookups refuse instead of guess, and sanity-check every mask. Making the fingertip usable: raw hand tracking jitters, and your own finger ends up in the frame. We added adaptive filtering, a pinch commit gesture and live tuning controls. Model quirks: Gemini’s segmentation masks were empty or too slow, so Gemini finds the box and MediaPipe or SAM cuts the mask.

Accomplishments that we're proud of

Point, speak, and watch a real object change, end to end, built in one day by Claude and Rahul.

What I learned

You need to let an idea snowball. This started as a project where I wanted to help animators by letting them point/select objects, move them around, and explain what they want to happen to an AI, which then goes and does the actual animating. I talked to some friends and they said to pick another option that wouldn't be AI-slop, but I kept thinking about how I wanted to see the idea through. As I was having Claude setup the repo I had the idea of involving the Meta glasses, and that turned into this project submission.

What's next for Reality is Yours

  • Run on Ray-Ban Meta glasses: a thin phone relay app using Meta’s Wearables Device Access Toolkit to stream the glasses’ camera and mic into our pipeline. Meta’s current toolkit doesn’t let apps draw onto standard glasses, so the edited view appears on a phone or screen first, and in the wearer’s eyes once display glasses allow it.
  • Once there's an actual AR component to the glasses, the potential for this project skyrockets since we can add live objects that the wearer can also see in real-time (not just the viewers). Allows for creating, designing, and literally the opportunity to let your imagination run wild. Also multiplayer features by joining other people's Meta worlds, see what they see, add to their worlds, like a Minecraft server!

Built With

Share this project:

Updates

Submission history