Welwi — Your Eyes, Your World

Inspiration

Visually impaired users already have AI that lets them see — point a phone at a room and something will describe it, point it at a sign and something will read it. That problem is essentially solved. But seeing isn't the same as living. None of those tools remember your day, keep your notes, manage your calendar, or actually do anything with what they just saw — there's no manager sitting behind the eyes. We kept coming back to one question: what if the assistant didn't just narrate the world back to you, but actually acted in it — took the note, made the calendar edit, reminded you before you had to ask? That's the gap Welwi is built for.

What it does

Welwi is a voice-first, multi-agent companion built on Gemini and Google's Agent Development Kit. Open the app and it's already listening — no login, no home screen, no menu to tap through. From there, in one continuous conversation, it can:

• Take notes by voice — propose, confirm, save — and read them back, edit them, or delete them later.
• Manage a real calendar the same way: create events, reschedule them, list the whole day back to you, all through natural language.
• Turn on its own camera only when it actually needs to look at something — read a poster on a wall and propose adding the event straight to your calendar, no typing involved.
• Speak up on its own. When a scheduled event's time arrives, Welwi interrupts with a real, spoken reminder — unprompted, in its own voice — instead of a silent push notification you have to notice and unlock your phone to read.

Every action follows the same trust pattern: the agent proposes what it's about to do, says it out loud, and only acts after you confirm. Nothing happens silently.

How we built it

The backend is a genuine multi-agent system, built with Google's Agent Development Kit (ADK) and deployed on Cloud Run as two services, five agents total:

• welwi_orchestrator — the root text agent. Runs on gemini-3.6-flash, Google's newest flash model, and does nothing but route: it reads the request and hands it off to whichever specialist can actually act on it, in a single turn, with no "let me check" round-trip.
• note_agent (gemini-3.6-flash) — owns the propose → confirm → save/list/edit/delete lifecycle for notes and undated reminders.
• calendar_agent (gemini-3.6-flash) — owns every calendar write: create, reschedule, delete, and "what's on my day."
• iris_agent (gemini-3.6-flash) — Welwi's vision specialist, invoked the instant a message carries an image; reads a flyer or poster and proposes the calendar entry directly.
• welwi-voice-agent — a standing Gemini Live session on gemini-3.1-flash-live-preview, holding one continuous audio/video connection with the full toolbelt available at every turn, rather than negotiating handoffs mid-conversation the way the text side's four agents do.

We picked gemini-3.6-flash specifically for the orchestrator and every sub-agent because routing and tool-calling are latency-sensitive and happen on every single turn — it's fast enough to keep the propose/confirm rhythm feeling instant, while still reliably picking the right specialist out of four. Both services share the same Firestore-backed tool layer — a note or event created by voice shows up identically through the text path and vice versa. The Flutter client streams microphone audio and, on demand, camera frames over a WebSocket to the Live agent, plays back Gemini's own synthesized voice directly (not a device text-to-speech layer), and mirrors every confirmed action into local state so a sighted companion can glance at the same data the agent just wrote.

Challenges we ran into

Almost every hard bug here was a real, evidence-based debugging story, not a guess:

• The agent would answer once, then go silent. Root cause: Cloud Run and the underlying Gemini Live session both have a finite connection lifetime, and nothing was reconnecting when that session naturally ended. Fixed with automatic, silent reconnection the instant the live connection drops, for any reason.
• Choppy, word-by-word audio. We were handing every small network chunk of audio straight to the platform's audio player, and each play() call has real setup overhead — that overhead was the stutter. Fixed by batching chunks into fewer, larger playback calls.
• The agent seemed to answer the same thing multiple times. This turned out to be two separate, real causes stacked on top of each other: first, a missing Android audio-focus configuration meant playing the agent's voice silently muted the microphone, breaking the next turn; second — and the deeper one — the phone's own microphone was picking up its own speaker output, and Gemini's server-side interruption detection read that echo as the user talking over the agent. The fix was proper acoustic echo cancellation on the recording side, not fighting the symptom on playback.
• "What's on my calendar" came back empty, even with real events on the calendar. Traced to a genuinely interesting bug: the agent's session state never actually captured the real user id from the live connection, so every write silently fell back to a default bucket — meaning a naive re-seed of "the same" data under the id we assumed was in use was invisible to the agent that only ever looked at the fallback bucket. Confirmed by directly inspecting what the deployed tool code was actually keying on, not by guessing.
• A single missing Firestore index turned "list my events" into a request that crashed the entire live session, not just that one tool call — because the exception wasn't caught narrowly enough to degrade gracefully.

Accomplishments that we're proud of

Every one of the fixes above was root-caused with real evidence from the actual deployed production services — not assumed and shipped. The result is an agent that holds a genuinely natural, interruptible, multi-turn voice conversation while taking real, durable actions across notes, calendar, and vision, with zero taps required to start.

What we learned

The gap between "the agent has the right tool" and "the agent reliably chooses the right tool in every phrasing" is real — we had to explicitly disambiguate between reading notes and reading a calendar in the instruction, even though the tools themselves were correct all along. And live, bidirectional voice is a fundamentally different engineering problem from turn-based chat: echo, interruption, and reconnection aren't edge cases, they're the normal shape of a real conversation, and each one needed a deliberate design decision rather than a default.

What's next for Welwi

Wiring the calendar tools back to the real Google Calendar API (the integration exists in code; only the OAuth token minting is left), and closing the loop on true session resumption so a natural reconnect doesn't lose conversational context mid-sentence.

Built With

• agentdevelopmentkit(adk)
• cloudrun
• dart
• flutter
• gemini
• google