Glance

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Glance

Inspiration 2.2 billion people worldwide live with a vision impairment, at least 1 billion of those cases preventable or unaddressed (WHO, World Report on Vision, 2019). Most camera-based assistants default to "describe everything," which is impressive in a demo and exhausting to live with — and usually needs the cloud. The "photo + question → short answer" pattern was validated by real blind users as far back as Bigham et al.'s VizWiz (UIST 2010); what's new is that Gemma 3n finally makes that pattern runnable entirely on a phone (2–3GB memory footprint). We built the narrow, honest version of this idea instead of another generic vision-AI demo.

What it does Native iPhone app — tap or speak, point the camera, ask one precise question, get one short spoken answer. 6 core tasks: read a page/menu/sign, identify currency, read expiration dates, read prescription labels, identify an object or color — plus follow-up questions on the same frame. 2 Phase 2 stretch features: ad-hoc scene description, and navigation (vision-guided nudges or real MapKit directions). Zero network calls except the one disclosed navigation exception.

How we built it Native SwiftUI, split into modules against a shared contract (Engine/Contracts.swift) so three people built in parallel. GlanceGemmaModelClient wraps Google's LiteRT-LM running Gemma 3n E2B, GPU-backed. PromptBuilder.swift routes every intent through one model with task-specific prompts. ResponseParser.swift is honesty-first — an UNCLEAR token plus a bounded uncertainty check trigger a fixed honest fallback instead of a guess. GuidanceSession/MapNavigator handle Phase 2. VoiceOver + haptics fire at every state transition. 18 unit tests plus real on-device integration tests back it up.

Challenges De-risking on-device inference first (before any app integration); tuning honesty-detection so it doesn't discard legitimately hedged-but-useful answers; coordinating three parallel builders against an evolving contract; and a mid-build scope debate about adding navigation, resolved by keeping it Phase 2 and clearly separating the no-network vision-guided version from the one disclosed network exception (real MapKit routing).

Accomplishments A real, tested, working on-device pipeline (not a mockup); an honesty-first failure path that's actually enforced in code and tests; full accessibility from the first UI commit; a pitch grounded in cited research, with equal honesty about what's a scoped hackathon cut vs. finished.

What we learned On-device multimodal LLMs are genuinely viable for accessibility tools today; narrow scope reads as more trustworthy than "describe everything"; defining the shared contract before any feature code is what let three people build in parallel at all.

What's next Live GPS-triggered turn-by-turn (currently speaks the full route upfront); broader real-world accuracy testing; actual usability testing with blind/low-vision participants (everything so far is fixture-validated, not user-validated); evaluating MediaPipe LLM Inference API as an alternative runtime; a smaller quantized model if latency needs it.

Built With

• avfoundation
• avspeechsynthesizer
• core-location
• gemma-3n
• ios
• litert-lm
• mapkit
• metal
• speech-framework
• swift
• swiftui
• xctest