Voidpilot

Inspiration

Most AI products still feel like text boxes. You type, wait, read, and then repeat. We wanted to build something that feels more like a live creative partner: something you can speak to naturally, explore ideas with in real time, and watch generate useful outputs while the conversation is still happening. We were excited to test the waters of multi agent orchestration through Gemini Live.

That led to Voidpilot: a voice-first multimodal Gemini experience that treats conversation as the interface. Instead of separating chat, research, generation, and output into different tools, Voidpilot brings them together into one fluid flow. The goal was to make Gemini feel less like software you operate and more like an active collaborator that can listen, respond, ground itself in context, and create alongside you.

What it does

Voidpilot is a web-based multimodal AI assistant built around Gemini.

It supports several experiences:

Live Mode for real-time voice conversation
Brainstorm Mode / Open Studio for voice-driven ideation with generated markdown artifacts, images, and videos
Creative Spark for fast visual brainstorming with a streamlined gallery-first experience
Walkthrough Mode for grounded, voice-guided project exploration

The app streams audio in real time, transcribes both the user and Gemini responses, and keeps the interaction low-friction and conversational. In brainstorm flows, users can talk naturally while the system generates structured documents, images, and videos in the background. In walkthrough mode, answers are grounded through a server-side project context search flow so the assistant explains the project based on retrieved context rather than unsupported guesses.

How we built it

We built Voidpilot as a full-stack web app:

Backend: FastAPI + WebSockets + Python
Frontend: React 19 + Vite 7 + TypeScript + Tailwind v4 + shadcn/ui + Framer Motion
AI integration: Google GenAI SDK and Gemini Live API
Persistence / sharing: Firebase Auth, Firestore, and Cloud Storage
Deployment: Google Cloud Run

At the core of the app is a shared Gemini Live session wrapper that manages realtime audio/text streaming, transcription, tool execution, session resumption, and context window handling. The frontend captures mic input, streams audio to the backend, and plays Gemini audio responses back in the browser.

For multimodal generation, the brainstorm workflows use:

Gemini Live for conversation
Gemini 3.1 Flash Image for image generation
Gemini 3.1 Flash Lite / Gemini Flash / Gemini 3.1 Pro for delegated text work and prompt enhancement
Veo 3.1 for video generation

For a grounded explanation of the Voidpilot codebase, Walkthrough mode uses a server-side search_project_context tool backed by Gemini File Search using a delegated Gemini 3 Flash agent.

Challenges we ran into

One of the biggest challenges was making realtime interaction feel smooth instead of fragile. Voice apps are unforgiving: even small delays, interruptions, or state mismatches break the illusion of a natural conversation and cause hard cashes or unwanted behavior.

Some of the hardest parts were:

keeping audio streaming, playback, and transcription synchronized
handling reconnects and session resumption cleanly
preserving conversation flow while tool calls run in the background
grounding walkthrough answers without slowing the live experience too much
coordinating multiple generation paths across voice, text, image, and video
keeping the frontend and backend message formats aligned as the product evolved
dealing with the intricacies of the Gemini live api as its not used by other people as much as text APIs.

Balancing responsiveness with richer capabilities was a constant challenge.

Accomplishments that we're proud of

We’re proud that Voidpilot is not just a demo of one API call — it’s a full product experience built around Gemini with agent orchestration, tool calling, and a plethora of Gemini sub-integrations, along with a full auth and storage system.

Highlights we’re especially proud of:

a voice-first Gemini experience that feels live and conversational
realtime transcription for both sides of the interaction
multimodal creation inside the same session: text, images, and video
a distinct Creative Spark mode with a focused visual-generation UX
grounded Walkthrough Mode backed by project context retrieval
session resumption and persistence so conversations can continue across reconnects
public share links for brainstorm sessions and generated artifacts
a working public deployment on Cloud Run

We’re also proud that the product feels opinionated. It is not trying to be a generic chatbot wrapper — it is designed around live collaboration.

What we learned

We learned that building with live multimodal models is as much about interaction design as model capability.

A few major takeaways:

voice changes the product completely; latency and turn-taking matter as much as raw model quality
tool use has to feel invisible to the user, or it breaks conversational momentum
grounding is essential when the product is meant to explain a real codebase or project
multimodal generation becomes much more compelling when it happens inside a live conversation rather than as a separate workflow
product clarity matters: giving users distinct modes with different constraints creates a much better experience than one overloaded interface

We also learned how important it is to design the full system around Gemini, not just attach Gemini to an existing app.

What's next for Voidpilot

Next, we want to push Voidpilot further as a true live multimodal agent.

Areas we want to expand include:

deeper multimodal workflows across voice, visuals, and structured artifacts
stronger retrieval and grounding for walkthrough-style experiences
richer collaborative generation loops for brainstorming and storytelling
more seamless session portability and persistence
broader sharing and publishing workflows for generated outputs
continued refinement of the real-time UX so the assistant feels even more natural and responsive
Support for Android XR with threeJS artifacts so you can talk to Gemini Live and watch your world come alive around you.

The long-term vision is simple: make interacting with Gemini feel less like using software and more like collaborating with a living creative system.