Inspiration

Learning to drive involves a lot more than following directions. You're constantly interpreting what's happening around you, anticipating hazards, and making decisions in unfamiliar situations. Having an experienced instructor in the passenger seat makes a huge difference, especially when they can point out something you missed or explain a situation as it happens.

But most learner drivers spend the majority of their practice hours with a parent or guardian, not a professional instructor. Parents aren't necessarily trained to provide calm, consistent feedback, and private lessons can be expensive.

We wanted to explore whether AI could help bridge that gap.

Existing driving technology is great at navigation, recording footage, and analyzing driving behavior afterward. But learning to drive is also about understanding why certain decisions matter, and receiving guidance while you're still in the situation.

That's why we built Voya, a conversational AI driving companion that combines real-time road awareness with natural, expressive voice interaction.

Our goal was to create something that feels less like another screen demanding your attention and more like a knowledgeable companion in the passenger seat.

What it does

Voya turns what's happening on the road into a conversation.

Using a phone's camera and GPS, Voya recognizes relevant objects in the driving environment and tracks upcoming road features. It uses this information to provide proactive, contextual coaching through voice.

Unlike a conventional navigation app, Voya can also listen and respond. Learners can ask driving-related questions, request clarification, or repeat previous guidance without having to type or navigate a screen.

Key features include:

Proactive voice coaching: Provides spoken reminders about pedestrians, cyclists, intersections, and other relevant driving situations.
Two-way conversations: Uses Cartesia Ink for speech recognition and Sonic for natural spoken responses.
Expressive speech: Dynamically adjusts emotion, speed, and volume to suit the content of each response.
Context-aware navigation: Combines GPS and OpenStreetMap data to provide timely reminders about upcoming turns, stop signs, and crosswalks.
Intelligent speech prioritization: Manages interruptions, prevents overlapping messages, and avoids outdated or repetitive coaching.
Live and replay modes: Supports live camera input and recorded driving sessions synchronized with GPS.
Accessibility and privacy: Provides captions, microphone controls, and a data-use panel showing what information leaves the device.

Voya is designed to supplement supervised driving practice, not replace a licensed instructor or the driver's judgment.

How we built it

We built Voya around three main capabilities: understanding the environment, deciding what information matters, and communicating it naturally through voice.

Voice: Cartesia Sonic and Ink

Voice was central to Voya's design. Driving is an environment where low latency and expressive communication are especially important. A reminder delivered too late might no longer be useful, and the tone of a response can change how a nervous learner receives it.

We integrated Cartesia Sonic 3.6 for speech synthesis and Cartesia Ink 2 for speech recognition. When a learner asks a question, Ink transcribes it, Claude generates an answer using available driving context, and Sonic speaks the response.

One feature we particularly focused on was dynamic voice delivery.

Rather than using the same tone for every sentence, Claude selects from 11 supported emotions, including calm, confident, determined, sympathetic, and curious. It also chooses the speed and volume of each response, which we pass directly into Cartesia's generation_config.

For example, a reminder approaching a stop sign can use a firmer, more noticeable delivery, while reassurance for a nervous learner can be softer and slower. This allows Voya to maintain a consistent voice while adapting how it speaks.

Latency was another major focus. We used Cartesia's streaming WebSocket API to play speech incrementally instead of waiting for a complete audio file. We also implemented connection warming, audio cancellation, and interruption handling.

For added reliability, we generated six fallback audio clips using Cartesia Sonic, allowing selected reminders to play immediately even when a live speech request is unavailable.

We also used short-lived access tokens to keep our Cartesia API key off the device.

Ultimately, we weren't simply building text-to-speech. We needed Voya to decide what to say, how to say it, and when to speak.

Computer Vision: MediaPipe and Claude

We used MediaPipe's EfficientDet-Lite0 for on-device object detection, identifying vehicles, pedestrians, cyclists, and traffic signs. Lightweight tracking helps retain context across consecutive frames.

The detections are converted into structured descriptions containing object categories, approximate positions, confidence scores, and apparent movement. Rather than continuously uploading video, Voya sends these descriptions to Anthropic Claude.

Claude uses an instructor-style coaching prompt and recent observations to determine whether there's something useful to say, or whether staying silent is the better choice.

We also designed the system to account for uncertainty. Recognizing a car or stop sign doesn't necessarily establish vehicle intent or right-of-way, so Voya avoids making unsupported assumptions.

Mapping: OpenStreetMap and GPS

We wanted Voya to understand not just what's visible, but also what's coming up.

Using OpenStreetMap, OSRM, and the Overpass API, we built a route-processing pipeline that identifies turns, stop signs, crosswalks, and traffic signals along a driving route.

For recorded sessions, GPS timestamps are synchronized with the video timeline. For live sessions, Voya uses the device's GPS location. The system matches positions to the route, estimates distance and time to upcoming features, and generates appropriately timed reminders.

We also account for GPS inaccuracies, off-route positions, and implausible location jumps to reduce incorrect guidance.

These reminders aren't hardcoded into the demo footage. They're generated from route information and synchronized with the driver's progress.

The mapping system also provides context for conversational responses, allowing Voya to answer questions about upcoming road features and recent trip information.

Frontend and Architecture

We built Voya with React, TypeScript, and Vite, deployed through Vercel.

The vision, mapping, reasoning, and voice modules communicate through a shared event-driven architecture. A central speech arbiter manages message priorities, expiration, interruption, and duplicate suppression.

This is particularly important when several things happen at once. A learner might ask a question while a route reminder is queued, or Claude might return coaching about a situation that has already passed.

Our system prioritizes requested answers, prevents competing audio from overlapping, and discards stale information.

We also built an interface displaying driving footage, visual detections, route information, captions, and a timeline explaining Voya's decisions.

Challenges we ran into

Latency

One of our biggest challenges was coordinating multiple AI systems with different response times. A driving observation can become outdated while waiting for visual processing, Claude inference, and speech generation.

We addressed this through local object detection, bounded analysis requests, stale-response filtering, and Cartesia's streaming audio generation. Connection warming and pre-generated fallback clips further help reduce delays.

We learned that responsiveness depends on the entire pipeline, not just the speed of an individual model.

Voice Interaction

We initially thought of speech as the final stage: generate a response, then read it aloud. But making Voya conversational required much more.

Should an explanation sound confident or reassuring? How should its volume change? When should Voya speak without being asked? What happens when an important message arrives during playback?

We experimented with Cartesia's expressive speech controls and built audio orchestration to manage delivery, interruptions, and priorities.

The timing, tone, and pacing of speech turned out to be just as important as the words themselves.

Multimodal Reasoning

Computer vision and mapping provide complementary information, but neither offers a complete understanding of a driving situation.

We had to distinguish reliable observations from assumptions, preserve uncertainty, and prevent incomplete information from becoming overly confident coaching.

This was especially important because our application deals with real-world situations where incorrect advice could have consequences.

Integration

With four team members developing voice, vision, mapping, and application integration separately, coordinating interfaces and timing was another major challenge.

A shared event architecture and careful state management helped us handle replay synchronization, session resets, microphone state, and overlapping audio.

Accomplishments we're proud of

Two-way conversational driving assistance powered by Cartesia Sonic, Ink, and Claude.
Expressive, context-dependent speech with 11 supported emotions and dynamic speed and volume adjustments.
Low-latency streaming architecture with connection warming, speech cancellation, and fallback audio.
Hybrid computer vision combining local object detection, tracking, and Claude-based reasoning.
GPS-synchronized route guidance using OpenStreetMap, OSRM, and Overpass rather than scripted alerts.
Event-driven orchestration coordinating multiple AI systems and prioritizing spoken information.
Privacy-conscious design that keeps camera frames on the device, uses short-lived API tokens, and provides transparent data-use information.

We're especially proud that these components work together as one experience rather than feeling like separate AI features.

What we learned

Building Voya taught us that integrating computer vision, language models, mapping, and speech is about much more than connecting APIs. Each system introduces different timing requirements, limitations, and uncertainties.

Our biggest takeaway was how much thought goes into designing an effective voice interface.

Natural-sounding speech is only part of the equation. The assistant also needs to know when to speak, how to adjust its delivery, and when silence is more useful.

Working with Cartesia let us explore these challenges through streaming audio, expressive voice controls, and real-time speech management.

We also learned the importance of grounding AI-generated explanations in reliable observations, particularly when the application involves real-world decisions.

What's next for Voya

We'd love to make Voya's coaching more personalized, adapting its guidance, frequency, and speaking style as learners gain experience.

We also want to expand post-drive conversations, allowing users to review confusing situations, ask questions about their sessions, and identify what to practice next.

On the technical side, we're interested in improving temporal scene understanding, recognizing more complex road interactions, and making speech even more responsive. Broader real-world use would also require validation with qualified driving instructors and testing for accuracy and distraction.

Ultimately, we see opportunities for conversational AI in many hands-on activities where people benefit from real-time guidance.

Voya explores what happens when AI can not only interpret the world around you, but communicate with you naturally while you interact with it.

Try Voya: Live Demo

Source Code: GitHub Repository

Built at Cal Hacks 13.0 by Jeev, Tejas, Nithila, and Rohit.