Gemini Multimodal Agent Platform

• 

• 

• 

Inspiration

What it does## Inspiration

Google's Gemini Live Agent Challenge asks developers to go beyond standard text chatbots and build agents that can see, hear, speak, and act in real time. That challenge shaped our project directly.

Instead of building a single narrow demo, we wanted to create one production-style multimodal platform that covers all three challenge categories in a unified architecture.

What the project does

Our project combines all three Gemini Live Agent Challenge tracks in one system:

• Live Agent for realtime conversation, interruption handling, speech-driven translation, negotiation, and grounded research.
• Creative Storyteller for generating multi-scene stories that combine text, audio, image, and video into one narrative flow.
• UI Navigator for understanding interfaces, planning actions, and executing browser tasks with approvals and safety controls.

This means the same platform can talk with a user, generate a multimedia story, and safely interact with interfaces from one shared runtime.

How we built it

We built the project as a TypeScript/Node.js monorepo with a realtime gateway, an orchestrator, an API backend, a UI executor, and separate domain agents for live communication, storytelling, and UI navigation.

For the model layer, we use Gemini via the Google GenAI SDK.
For the cloud layer, the primary judged deployment path uses Google Cloud Run for the main services and Firestore for runtime state and approvals. We also use BigQuery and Cloud Monitoring to provide analytics and observability evidence.

The frontend is designed for judge-facing walkthroughs, while the backend exposes operator controls, approval flows, runtime health, and evidence export.

Transparency and evidence

A major requirement of the challenge is full transparency: no mockups, only real agent behavior and proof that the backend is running in the cloud.

To support that, our project produces repo-owned evidence such as:

• release and runtime proof artifacts,
• operator summaries and approval traces,
• session exports,
• visual evidence packs,
• and deployment/verification scripts for repeatable demos.

This lets judges inspect not only what the agent says, but also how it runs, what decisions were gated, and whether the cloud runtime is real and observable.

What we learned

We learned that next-generation multimodal agents require much more than prompting. They need orchestration, interruption handling, approval logic, replayable evidence, and cloud observability.

We also learned that:

• realtime agents need reliable session and interruption management,
• storytelling needs structured scene and media pipelines,
• UI agents need clear safety gates before sensitive actions,
• and demos become much stronger when every claim is backed by visible runtime proof.

Challenges we faced

The hardest challenge was combining three very different agent categories into one coherent product without turning it into three unrelated demos.

Another challenge was keeping the experience short and judge-friendly. Because the challenge demo must stay under four minutes, we had to design flows that show live interaction, storytelling, UI execution, and cloud proof without wasting time.

We also had to solve transparency and safety at the same time: proving real cloud execution, while pausing risky UI actions for human approval and keeping the system understandable for judges.

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Gemini Multimodal Agent Platform

Built With

• bigquery
• cloud-monitoring
• css
• firestore
• gemini
• github-actions
• google-cloud-run
• google-genai-sdk-(@google/genai)
• html
• javascript
• node.js
• rest-apis
• typescript
• websockets-(ws)