Vantage

Inspiration

Phones are everywhere. Many mobility aids are expensive, loud, or built for sighted screens. We wanted a chest-worn companion that feels like another sense: sparse haptics when something is close, and a calm voice when you ask — not a map and not a replacement for a cane or guide dog.

What it does

Vantage is a voice-first iOS guide you wear on a lanyard. It has two modes.

Freeroam — walk freely and feel what's around you.

• LiDAR (or camera) watches left / ahead / right. Near, medium, and far haptic bands; continuous buzz when something is very close. No beeps, no side bars.
• Sparse callouts — on-device people and object recognition, spoken only as you approach, with cooldowns. Dodge directions only while you're walking.
• Take me back — retraces your path to where you started using ARKit tracking alone. No network.

Navigate — say where you want to go ("the exit", "room 214").

• Gemini picks one landmark in view and marks where it is in the photo; LiDAR pins it as a fixed point in the room, so it stays put when you turn.
• A 3D spatial tone sits on the target (and follows your head with AirPods), with distance callouts at 20, 15, 10, 6, and 3 ft.
• Reach a landmark, walk past it, or stall, and Vantage asks for the next one automatically.
• Sign reading on device — spots a sign matching your destination and locks onto it, even offline.
• Live room mesh — maps walls and doors as you walk: "Vantage, where's the door?" answers instantly, and the beacon routes around obstacles in the straight line ("Something's in the way. Head left.").

Both modes

• Ask on demand — Say "Vantage…" then a question ("What's ahead?", "What's around me?", "Repeat"). A chime and glow confirm it heard you. Answers stay short and use the phone's measured distances, not guesses; Vantage says when it isn't sure.
• Night vision — LiDAR depth overlay in Color or 3D Spatial shading, out to 10 m. Switches to full-screen depth automatically in the dark. Voice toggle works offline.
• Offline when it matters — Scene answers try Gemini online, then on-device Gemma, then simple Vision templates. Safety depth never waits on the network.
• Calm voice — ElevenLabs with saved clips for common lines (instant and offline), system voice fallback, loud enough for a phone speaker.

Vantage is not a GPS navigator and not a cane/dog replacement.

How we built it

SwiftUI + ARKit on one rule: safety never waits on the network.

• Depth into left / center / right zones over a chest-height band (floor filtered out)
• Core Haptics for Freeroam's primary cue
• Apple Speech + wake word for hands-free ask
• Gemini for online scene Q&A and landmark picks, fed the phone's sensor readings; Gemma (LiteRT-LM) on device when offline
• LiDAR depth turns a landmark in a photo into a world-anchored 3D point; HRTF spatial audio with AirPods head tracking
• ARKit scene reconstruction for a live room mesh (walls, doors), with an occupancy grid and A* to steer around obstacles
• Apple Vision text recognition for on-device sign reading
• High-contrast ink / teal / cream UI with huge mode cards and VoiceOver-friendly targets; talks from launch

Challenges we ran into

• Keeping cues sparse — stacking beeps + haptics + talk overwhelms people
• Depth buffer orientation vs portrait UI (left/right flipping), and mapping a point in a portrait photo back into 3D
• Wake-word reliability in a noisy hall without hearing itself as a command
• Keeping ARKit frames flowing while AI, TTS, sign reading, and meshing all run
• Cloud model latency swings — Navigate keeps tracking the pinned target on device between calls

Accomplishments that we're proud of

• Freeroam that you can feel without looking at the phone
• A Navigate beacon that stays on the door when you turn around
• Voice-from-launch UX so a blind user isn't stuck on a silent menu
• On-device Gemma, sign reading, and room mesh when the venue Wi-Fi dies
• A restrained design: one job per screen, big targets, calm motion

What we learned

Depth decides where; models decide what. Accessibility is mostly restraint.

What's next

• Field-test Navigate in real buildings with cane and guide-dog users
• On-device Gemma for Navigate steps when offline
• Stairs and drop-off warnings; harden outdoor crossings

Vantage is a 24-hour prototype that adds another sense to a phone people already carry.

Built With

• arkit
• core-haptics
• core-motion
• elevenlabs
• gemma
• google-gemini
• ios
• lidar
• litert-lm
• speech
• swift
• swiftui
• vision
• xcodegen