Multimodal AI agent on Google Cloud using Gemini

Gemini can output text + images + narration in one response.

Comment

Inspiration Modern digital workflows generate massive amounts of information across dashboards, conversations, documents, and visual interfaces. Security analysts, students, and professionals often switch between tools to interpret alerts, create reports, and take action. Traditional AI assistants rely mainly on text input, limiting real-world usability. We wanted to build an AI agent that interacts more naturally — one that can see, listen, understand context, and respond using multiple forms of media. Inspired by real-world operational environments such as security operations centers and complex digital workflows, we designed a multimodal agent capable of real-time reasoning and interaction using Google’s Gemini models. The goal was to move beyond chatbots and demonstrate how AI agents can become active collaborators rather than passive responders. What it does The Multimodal AI Agent is a next-generation assistant built on Google Cloud that combines vision, voice, and reasoning capabilities. The agent can: Analyze screenshots or visual dashboards using Gemini’s multimodal understanding Accept voice or text instructions from users Generate structured explanations and actionable recommendations Convert responses into natural speech Understand UI screens and suggest or execute actions Create multimedia content including stories, visuals, and narrated outputs Key capabilities include: Live Interaction — users can speak naturally and receive spoken responses Visual Understanding — interprets images, interfaces, and diagrams UI Navigation — understands on-screen elements for workflow automation Creative Storytelling — produces interleaved text, images, and narration The system transforms AI from a question-answering tool into an interactive multimodal agent. How we built it The project was developed using Google’s AI and cloud ecosystem. Core Technologies Gemini 1.5 Pro for multimodal reasoning Google GenAI SDK & Agent Development Kit (ADK) for agent orchestration Google Cloud Run for hosting Cloud Storage for media handling Speech-to-Text & Text-to-Speech APIs for live voice interaction Architecture User provides input via voice, text, or image. Inputs are processed and routed through an agent orchestrator. Gemini analyzes multimodal context. The agent selects tools (analysis, storytelling, automation). Outputs are generated as text, audio, or multimedia responses. Results are stored and served using Google Cloud services. Components Built Multimodal reasoning engine Live voice interaction pipeline Screen understanding module Story generation pipeline Cloud deployment infrastructure Challenges we ran into Multimodal orchestration: Combining voice, image, and reasoning workflows required careful input routing and context management. Latency in real-time interaction: Streaming audio while maintaining responsiveness demanded optimization. Prompt design: Ensuring Gemini produced structured, actionable outputs instead of generic responses required iterative prompt engineering. UI interpretation ambiguity: Screenshots lack DOM information, so visual reasoning had to compensate using contextual prompts. Media synchronization: Aligning generated images, narration, and storytelling outputs into a coherent experience was technically complex.

Built With

  • 2.0
  • ai
  • api
  • assets
  • cloud
  • firebase
  • flash
  • gateway
  • gemini
  • generated
  • google
  • multimodal/interleaved
  • native
  • next.js
  • output
  • storage
  • storing
  • vertex
  • with
Share this project:

Updates