Vyze - Offline AI Vision Assistant

Inspiration

My drive to build Vyze comes directly from my brother, who is blind. Living alongside him, I witness daily the practical friction and challenges visually impaired individuals encounter when navigating a world designed entirely for sight.

Existing assistive tools often fall short due to constant internet dependence, expensive subscription fees, high audio latency, complex hardware requirements, or imprecise touch interfaces. Watching these everyday struggles firsthand inspired me to build Vyze—a fully offline, privacy-first spatial vision assistant designed to serve as a fast, reliable pair of eyes right from a smartphone.

What it does

Vyze is a 100% on-device spatial AI vision assistant. Powered by Gemma running locally via Google's LiteRT-LM framework, Vyze provides immediate spatial feedback and object identification without requiring internet access or subscription fees. Key capabilities include:

Spatial Audio Guidance: Uses a software-based $3 \times 3$ virtual grid overlay to translate camera preview frames into spoken directions (e.g., "Cup in the top-right of your frame").
Intent-Based Region Cropping: Enables users to tap near an area of interest to crop local regions for targeted optical character recognition (OCR) and visual language analysis.
Layered Architecture: Routes fast local queries (color, light levels, text lines) through Kotlin-native fast-paths, reserving heavy VLM reasoning only when deep context is needed.

How we built it

Vyze was engineered natively for Android using a high-performance stack:

Core Framework & UI: Kotlin and Jetpack Compose for fluid UI states and accessible touch exploration.
AI & Vision Pipeline: Google's LiteRT-LM framework for local Gemma inference, combined with CameraX for real-time frame capture.
Fast Local ML: ML Kit for near-instant optical character recognition (OCR) and basic object/color segmentation.
Audio Feedback: Android System TTS and custom haptic feedback patterns to deliver zero-latency spatial orientation cues.

Challenges we ran into

Handling Touch Imprecision: Visually impaired users cannot pinpoint exact visual coordinates by sight. We solved this by developing an anchor-based bounding strategy where taps snap to nearby object contours rather than tight pixel coordinates.
On-Device Performance & Latency: Balancing deep multimodal vision processing with strict real-time feedback constraints on mobile hardware. Implementing layered tool routing ensured fast feedback loops while optimizing battery consumption.
Structured Context Translation: Converting raw pixel coordinates into structured, plain-text prompts (e.g., [Target location: Center-Right]) so Gemma receives spatial context naturally without complex coordinate processing.

Accomplishments that we're proud of

Zero Latency & 100% Offline Independence: Successfully running Gemma locally via LiteRT-LM to deliver complex spatial reasoning with zero network reliance or API cost.
Pure Software GIS: Eliminating the need for expensive external hardware sensors (like LiDAR or lasers) by using lightweight $3 \times 3$ grid algorithms in pure software.
Accessible-First Architecture: Creating a seamless fusion of tactile exploration, haptic cues, and spoken feedback designed specifically for screen-free use.

What we learned

We learned the critical importance of designing accessibility software around zero-latency feedback loops. By keeping processing entirely on-device, we were able to deliver consistent, offline performance that respects user privacy while removing reliance on external sensors or cloud infrastructure.