Inspiration
The inspiration behind Live Agent was to move beyond traditional text-based chatbots. Most AI systems respond only to typed prompts, which limits natural interaction. We wanted to create an AI that actively participates in the creative and development process.
By combining real-time vision, voice interaction, emotional intelligence, and live coding capabilities, we designed an AI companion that feels less like a tool and more like a collaborative partner sitting beside the user.
What it does
Live Agent is a multimodal AI companion designed with two powerful modes:
AI Emotional Companion A high emotional-intelligence assistant that detects the user’s mood through voice tone and visual cues. It provides empathetic conversations, encouragement, and emotional support to create a more human-like interaction.
AI Project Assistant (Tutor Mode) An intelligent coding partner that can see the user’s workspace (camera or screen). It helps build projects by generating HTML, CSS, and JavaScript code, creating AI-generated images, and updating a live preview workspace in real time based on verbal instructions.
This allows developers to build and modify projects simply by talking to the AI.
How we built it
Core AI: The system is powered by the Gemini 2.5 Flash Live API, enabling low-latency real-time processing of audio and visual inputs.
Frontend: The interface is built using React and Tailwind CSS, with Framer Motion used to create an immersive and responsive user experience.
Multimodal Integration: We implemented Web Audio API processors for streaming PCM audio and developed a high-frequency canvas-to-socket pipeline to transmit real-time visual data to the AI model.
Live Workspace: A dynamic iframe-based preview system allows the AI to instantly render generated code and images with hot-reloading capabilities, enabling users to see updates immediately.
Challenges we ran into
One of the biggest challenges was synchronizing real-time audio and vision streams while simultaneously handling AI tool calls for the workspace.
The system needed to allow the AI to:
understand a spoken instruction,
interpret visual input,
generate code or images,
and update the live preview — all without interrupting the conversation.
Achieving this required careful state management, optimized socket communication, and strong error handling for multimodal data streams.
Accomplishments that we're proud of
We are especially proud of the Tutor Mode capability.
The AI can listen to a user’s request, analyze the workspace, generate an image using the generate_image tool, and automatically write the necessary CSS and HTML to integrate it into the project — all within seconds.
This demonstrates a new way of interacting with development tools and highlights the future potential of AI-assisted programming environments.
What we learned
Through building Live Agent, we gained hands-on experience with:
The @google/genai SDK
Real-time PCM audio streaming
Multimodal AI interaction
Designing AI personas using system instructions
Managing tool-calling reliability in real-time AI systems
We also learned how to design user experiences that make AI feel more natural, responsive, and collaborative.
What's next for Live Agent
Our next goal is to expand Live Agent with Multi-user Collaboration, allowing multiple people to interact with the same AI agent within a shared workspace.
We also plan to introduce:
Multi-file project support
Integration with external documentation and developer APIs
Advanced autonomous coding capabilities
Ultimately, we aim to transform Live Agent into a fully collaborative AI development partner capable of assisting teams throughout the entire software development process.
Log in or sign up for Devpost to join the conversation.