NewYorker(Jarvis)

Inspiration

Navigating a new city is overwhelming. Millions of immigrants, tourists, and visually impaired transit riders face app fatigue and language barriers daily. We wanted to build an active, hands-free urban navigator that actually remembers your visual context instead of just analyzing a single static photo.

What it does

New Yorker is a real-time spatial AI companion. You point your phone's camera, speak naturally in any of 30+ languages, and get instant directional audio back in under 800ms. It can also autonomously check your transit balances and text you offline itineraries directly to iMessage.

How we built it

We wired a React/Vite frontend to a FastAPI Python backend. The Gemini API handles continuous multimodal frame processing, ElevenLabs streams the low-latency TTS, and MongoDB Atlas stores our rolling visual session memory. For agentic actions, we used Capital One Nessie (micropayments) and the Photon SDK (iMessage dispatch), all hosted on DigitalOcean and coded end-to-end in Cursor.

Challenges we ran into

Hitting our strict sub-second latency budget was incredibly difficult. We had to treat our perception-to-voice loop like flight software, ensuring graceful degradation and managing a continuous visual state machine so the AI wouldn't lose context between camera frames.

Accomplishments that we're proud of

We moved beyond standard LLM wrappers to build a true streaming WebSocket pipeline. We're incredibly proud of seamlessly merging live camera tracking, multilingual voice, and native OS actions (like sending formatted blue-bubble texts) into one fluid, hands-free experience.

What we learned

We learned that conversational spatial AI completely breaks down without persistent memory. Implementing a rolling visual context buffer was the breakthrough that made the system feel like a true companion rather than a simple Q&A bot.

What's next for NewYorker(Jarvis)

We plan to expand our agentic actions beyond transit, integrating deeper real-world micropayments and proactive detour routing for even more complex urban environments.