NavAI

Inspiration

NavAI reimagines navigation by integrating immersive AR with AI-driven natural language understanding and image analysis. With voice-activated queries and real-time AR map overlays, users can effortlessly discover points of interest and obtain contextual information—all without ever pulling out their phone.

What it does

Voice Processing - VoiceController.ts and SpeechToText.ts handle wake word detection ("Hey Alice") and voice commands

Map System — Multiple controllers (MapUIController, MapManipulation, etc.) manage the AR map experience

AI Integration — geminiHandler.ts coordinates between Gemini AI and the system, while PlacesWrapper.ts handles Places API interactions

Interaction — PinchController.ts and PopupController.ts manage user input and feedback Supporting Systems: Includes camera, voice input, and sound output configurations

How we built it

Hardware

Snap Spectacles — AR-enabled smart glasses

Built-in Camera & Mic — For seamless image capture and voice input

Location Sensors — Precise geolocation for dynamic mapping

Core Technologies

TypeScript — Strongly-typed development environment

Snap AR SDK — AR content rendering and spatial tracking

SIK (Spectacles Interaction Kit) — Gesture recognition and interaction

AI & APIs

Google Gemini AI — Natural language processing and visual analysis

Google Places API — Dynamic retrieval of nearby points of interest

Custom Flask Backend — Image processing and AI integration

Challenges we ran into

This hackathon was our first experience with the Spectacle ecosystem. This was an unique challenge given the novelty and breadth of the Lens Studio IDE and Spectacles hardware, requiring us to go to the limits of the documentation and beyond to achieve the features we wanted.

Accomplishments that we're proud of

We are incredibly proud of NavAI and its ability to bring people to the streets. Even while walking around the city demo-ing the product we learned new things about locations we once thought familiar.

What we learned

This hackathon was a fantastic learning experience for the whole team, picking up XR skills, learning the Spectacle ecosystem, as well as learning how to integrate various APIs with mobile and wearable systems.