GlassGPT

Inspiration

Smart glasses should feel like a natural extension of my mind, not another closed ecosystem. I asked myself: what if Meta hardware, OpenAI models, and Apple’s native capabilities could work together? GlassGPT explores that idea: an AI assistant with eyes, voice, and the ability to safely hand off real-world tasks to iOS.

What it does

GlassGPT is a real-time voice and vision assistant for Meta Ray-Ban glasses and iPhone. I can speak naturally through the glasses, ask what I am looking at, and hear answers back through the glasses speakers. With permission and confirmation, it can create reminders and calendar events, open Apple Maps directions, set timers and notifications, search contacts and hand off calls, create location-based reminders, play Apple Music, and save images to Photos. A Live Activity keeps the session visible on the Lock Screen and Dynamic Island.

How I built it

I built GlassGPT as a native SwiftUI iOS app. Meta Wearables DAT provides the glasses connection and camera frames; iOS Bluetooth audio routing uses the glasses microphone and speakers; and OpenAI’s gpt-realtime-2.1-mini powers the low-latency, multimodal conversation. To preserve bandwidth for natural voice interaction, I stream camera input at an extremely low 1 fps and send only the latest compressed frame when relevant to a request. I built a Realtime WebSocket client for PCM audio streaming, speech detection, interruption handling, vision-frame attachment, and response playback. I route native actions through a permission-gated iOS layer built with EventKit, MapKit, Core Location, MusicKit, Contacts, Photos, and local notifications.

Challenges I ran into

The hardest problem was balancing glasses video transport with Bluetooth HFP audio. Higher camera frame rates caused buffer pressure and unstable audio, so I iterated from live video to on-demand still capture and ultimately landed on a 1 fps low-quality stream that keeps vision responsive without compromising conversation. I also navigated Realtime API migration issues: deprecated endpoints, evolving GA session schemas, audio-format requirements, VAD behavior, and response interruption. Bluetooth hardware audio ran at 16 kHz while Realtime expects 24 kHz PCM, requiring explicit hardware-format capture and resampling. Finally, interruption was deceptively difficult. I had to stop queued playback, cancel active responses, and calculate truncation from confirmed played PCM frames, not wall-clock time, to avoid invalid audio truncation requests.

Accomplishments that I'm proud of

A genuinely hands-free assistant that can see through glasses and respond in real time. Stable barge-in behavior: I can speak over the assistant and it stops, listens, and handles my follow-up. Native Apple integrations with explicit user permission and confirmation. A privacy-conscious design: no continuous video is sent to the model, and sensitive features are opt-in. A polished iOS experience with a custom speaking animation, Liquid Glass controls, background voice support, Live Activities, Dynamic Island support, and debug audio mode.

What I learned

Building cross-platform AI experiences is as much about systems integration as it is about the model. Real-time AI requires careful handling of audio formats, transport bandwidth, interruption state, lifecycle transitions, permissions, and hardware constraints.

What's next for GlassGPT

Next, I want to expand the App Intents and Shortcuts surface so GlassGPT can become even more useful, and hopefully gives a glimpse to Meta / OpenAI / Apple execs of what could be built with the glass form factor.

Built With

• codex
• swift