-
-
Lore - point your camera, hear the story
-
-
Inspiration
I was standing in front of a building I'd walked past a hundred times and realized I had no idea what it was. I pulled out my phone, opened Google, typed a description, got a Wikipedia link, started reading... and the moment was gone.
There had to be a better way. Just point. Just ask. Just listen.
What it does
Lore listens to you, watches through your camera, and narrates what it sees — like a documentary, live, for whatever you're looking at. A monument. A dog. A book on your shelf. Ask a follow-up and it answers. Ask it to show you something and an image appears. Every fact it speaks is grounded in real search results, not guesses.
How we built it
The core is Gemini Live API running through Google ADK — a single bidirectional stream carrying audio, video frames, and tool calls simultaneously. On the backend, FastAPI handles the WebSocket connection and passes everything to an ADK agent that has two tools: Google Search for fact-grounding, and a custom image generation tool that produces visuals mid-narration. The whole thing runs on Cloud Run.
The frontend is React with raw Web Audio API — no libraries, just PCM audio capture and playback wired directly to the WebSocket.
Challenges we ran into
The streaming was brutal to get right. When audio, video, search calls, and image generation are all happening at the same time over one connection, the ordering matters in ways that aren't obvious until something breaks.
And getting Lore to actually sound like Lore — not like a search result being read aloud — took far more iteration than the code did.
Accomplishments that we're proud of
The interruption. Mid-sentence, you can ask a follow-up question and Lore stops, answers, and continues. It sounds obvious but making that feel natural — not jarring, not robotic — was genuinely hard.
What we learned
Voice changes everything. The same information that feels dry as text feels alive when it's spoken with pacing and narrative structure. Prompt engineering for voice is its own discipline.
What's next for Lore
Museum mode. City walks. Any place where there's something worth knowing about what's around you.
Log in or sign up for Devpost to join the conversation.