Live Case Companion

A real-time multimodal agent that helps users investigate clues, organize reasoning, and study through natural conversation on Google Cloud.

Comment

Inspiration

I wanted to build an AI agent that feels more like a live partner than a chatbot. Instead of typing into a text box, users can talk naturally, show real-world objects or homework through the camera, and continue the interaction over multiple sessions.

What it does

Live Case Companion is a real-time multimodal assistant built for investigation and study. In detective mode, it listens to the user, watches camera input, helps organize clues, updates reasoning, and saves case sessions for later continuation. In study mode, it supports interactive learning through voice-and-vision conversation, while also saving progress and follow-up analysis.

The system uses a two-role structure inside one experience. The live assistant handles real-time conversation, while a deeper analysis layer reviews saved evidence and generates summaries, insights, and next-step guidance for future sessions.

How I built it

I built the project with React and TypeScript using the official Google GenAI SDK for JavaScript (@google/genai). Real-time multimodal interaction is powered by the Gemini Live API, while follow-up analysis uses Gemini 3.1 Pro Preview. I also use Gemini TTS for voice playback of analysis results.

For the backend and persistence layer, I used Google Cloud and Firebase services, including Firebase Authentication, Cloud Firestore, and Google Cloud project hosting infrastructure. Session history, reasoning updates, and saved evidence are stored so the user can return and continue later.

Challenges I ran into

The biggest challenge was keeping the live multimodal experience stable over time. I had to work through issues around session continuity, audio handling, reconnection flow, and saving/resuming sessions cleanly without breaking the user experience.

Another challenge was balancing fast real-time interaction with deeper reasoning. I wanted the live assistant to respond quickly, while also supporting richer follow-up analysis after the session.

Accomplishments that I'm proud of

I’m proud that the app now supports real-time voice-and-vision interaction, session saving, session resume, and follow-up analysis in a single experience. I’m also happy that I was able to shape it into something that feels more like a companion than a standard chatbot.

What I learned

Through this project, I learned a lot about designing multimodal agent experiences, handling real-time audio/video interaction, structuring session memory, and connecting Gemini-based features with Firebase and Google Cloud services.

What's next for Live Case Companion

Next, I want to improve deployment robustness, expand the follow-up analysis pipeline, strengthen interruption handling, and make the experience even more natural for both investigation and educational use cases.

Built With

  • cloud-firestore
  • firebase-authentication
  • gemini-3.1-pro-preview
  • gemini-live-api
  • gemini-tts
  • google
  • google-genai-sdk-(@google/genai)
  • react
  • typescript
  • vite
Share this project:

Updates