SunSeekr Voice

Inspiration

Finding sunny outdoor spots in London is surprisingly hard. Weather apps tell you "it's sunny," but they can't tell you which pub terrace is actually getting sunlight right now — buildings cast moving shadows that change every hour. We built SunSeekr Voice so you can simply ask "find me a sunny pub near Kings Cross" and get a spoken answer with real-time data about which venues have sunshine on their outdoor areas.

What it does

SunSeekr Voice is a voice-first assistant that combines real-time building shadow analysis with natural voice conversation to help people find sunny outdoor spots to eat and drink:

• Ask naturally: "Find me a sunny pub near Shoreditch" or "What's the weather like?"
• Real sun data: Each venue shows what percentage of its outdoor area is in direct sunlight — not just whether it's "sunny" in the city
• Hourly sun charts: Visual bar charts show sun percentage hour-by-hour so you can plan when to visit
• Google Maps details: Ratings, reviews, photos, and walking directions for every venue
• Barge-in: Interrupt the agent mid-sentence to change your mind

How we built it

• Google ADK (Agent Development Kit) manages the Gemini Live session, tool orchestration, and audio streaming
• Gemini 2.5 Flash with native audio for real-time voice input/output — no separate STT/TTS pipeline
• Custom web frontend with WebSocket audio streaming (PCM 16-bit), gapless playback scheduling, connection state machine, and hourly sun chart components
• SunSeekr's sun data engine provides pre-computed building shadow analysis: each venue's outdoor area is mapped as a polygon, and sun/shadow percentages are computed hourly using the ShadeMap API
• Google Maps Platform (Places API + Directions API) for ratings, photos, and directions
• Deployed on Google Cloud Run with containerized ADK server

Challenges we ran into

• Gapless audio playback: PCM chunks from the Live API arrive asynchronously. We schedule each chunk using source.start(nextPlayTime) to eliminate gaps between audio segments.
• Agent tool-call reliability: Getting Gemini to consistently call get_venue_details for ALL returned venues (not just the first) required emphatic prompting plus a frontend fallback that auto-requests details for unenriched cards.
• Connection state management: Tracking 4 distinct states (idle → connecting → listening → responding) across WebSocket, microphone, and audio playback lifecycles.
• Barge-in handling: When the user speaks while the agent is responding, we reset the audio playback timeline and transition back to listening state.

Accomplishments we're proud of

• The sun chart visualization is genuinely useful — you can see at a glance when a venue gets sun and plan accordingly
• The voice experience feels natural and responsive, with visual state feedback matching what's happening
• Building shadow analysis is a unique data source that no other voice assistant has access to
• SunSeekr already has 250K+ users — this extends a real product, not just a hackathon prototype

What we learned

• Gemini's native audio mode eliminates the latency of separate STT→LLM→TTS pipelines
• Google ADK handles a lot of the session management complexity, letting us focus on the agent's personality and tools
• Visual feedback during voice interactions is crucial — users need to see what state the system is in

What's next

• Support for more cities beyond London
• Real-time sun tracking (live updates as shadows move)
• Group planning: "Find a sunny spot that works for 4 of us at 3pm"
• Mobile app integration with push notifications when your favourite spot gets sun

Built With

• fastapi
• gemini-2.5-flash
• google-adk
• google-cloud-run
• google-maps-platform
• javascript
• python