ARIA (Advanced Real-time Intelligence Agent)

ARIA is a multimodal AI agent that sees your world, hears your questions, and responds in real time.

Comment

The Story Behind ARIA

💡 Inspiration

We live in a world overflowing with information — images, voices, text, video — yet most AI tools still force you into a single narrow channel: type a message, get a message back. That always felt like a step backward from how humans actually communicate.

When the Gemini Live Agent Challenge was announced, it wasn't just an opportunity to build something cool — it was a call to rethink the entire interface between humans and AI. The question that drove us wasn't "what can we build?" but rather:

"What if AI could perceive the world the way we do — seeing, hearing, and understanding all at once?"

That question became ARIA.

The name itself is intentional. An aria in music is a self-contained piece for a solo voice — expressive, powerful, and complete on its own. That's exactly what we wanted this agent to be: a single, unified intelligence capable of handling any modality you throw at it, without switching tools, tabs, or contexts.

🔨 How We Built It

Architecture Overview

ARIA is built on a clean, stateless architecture optimized for real-time multimodal interaction:

User (Browser)
    │
    ▼
React Frontend  ──── Image (base64) + Text ──▶  Gemini Vision API
    │                                                   │
    ▼                                                   ▼
Cloud Run (Node.js)  ◀──────── Response ────────  Gemini 2.0 Flash
    │
    ▼
Firestore  (session & conversation history)

Multimodal Message Construction

The trickiest part of the build was correctly formatting conversation history for the Gemini API when messages alternate between text-only and image+text turns. A naive implementation breaks context. The solution was to reconstruct the full message array on every API call:

$$ M_n = {m_1, m_2, \ldots, m_{n-1}, m_n} $$

Where each message $m_i$ is defined as:

$$ m_i = \begin{cases} {role, \ text} & \text{if text-only turn} \ {role, \ [image_{base64}, \ text]} & \text{if multimodal turn} \end{cases} $$

This ensures the model always has full context — including what it previously saw — without any session stickiness on the server.

Stateless Scalability

Because all state lives in Firestore and is hydrated per request, the Cloud Run instances are fully stateless. This means horizontal scaling is trivially handled by Google Cloud:

$$ \text{Throughput} = \lambda \times \frac{1}{\mu} \quad \Rightarrow \quad \text{latency} \approx \frac{1}{\mu - \lambda} $$

Where $\lambda$ is the request arrival rate and $\mu$ is the processing rate per instance — as $\lambda$ grows, Cloud Run simply spins up more instances, keeping latency flat.

The UI Philosophy

We designed ARIA's interface around one principle: the experience is the product. Three deliberate choices defined the UI:

  1. Mode switching (Chat / Vision / Creative) without page reloads or context loss
  2. Quick-action prompts that surface automatically when an image is attached — reducing the cognitive load of multimodal interaction
  3. Animated waveform as a visual metaphor for "listening" — even before full voice integration

📚 What We Learned

Building ARIA taught us things no tutorial could:

1. Multimodal UX is a design problem, not just an API problem

Passing an image to Gemini is three lines of code. Making that feel natural and intuitive took days of iteration. The quick-action buttons were a late addition — and they changed everything.

2. Context is everything

A multimodal agent without memory is just a fancy image classifier. The moment ARIA could reference an image from three messages ago, it stopped feeling like a tool and started feeling like a collaborator.

3. Gemini's vision capability is deeply underestimated

We expected basic object recognition. What we got was nuanced scene interpretation, OCR, reasoning about spatial relationships, and even emotional tone detection from images. The model consistently surprised us.

4. Stateless ≠ memoryless

Designing for stateless Cloud Run instances forced us to think carefully about what "memory" means for an AI agent. The answer: memory lives in Firestore, not in the server. That separation of concerns made the system more robust, not less.

🧗 Challenges We Faced

Message Format Consistency

The Gemini API is strict about alternating user / assistant roles. A single misformatted turn causes the entire conversation to fail silently. We built a message normalization layer that sanitizes history before every API call.

Image Size & Latency

High-resolution images sent as raw base64 balllooned request sizes and introduced noticeable latency. We solved this with client-side canvas resizing before encoding:

$$ \text{Resize factor} = \min\left(1, \ \frac{W_{max}}{W_{original}}\right), \quad W_{max} = 1024 \text{px} $$

This kept image quality high while reducing payload sizes by up to $~70\%$ in practice.

Designing for Three Modalities at Once

Building a UI that gracefully handles text-only, image-only, and combined text+image turns — without separate input forms — required rethinking the entire input model. The solution was a unified composer that detects attached state and adapts its behavior accordingly.

🔮 What's Next

ARIA is a foundation, not a finish line. The roadmap:

  • 🎙️ Full voice I/O — Google Cloud Speech-to-Text + Text-to-Speech for a truly hands-free agent
  • 🖥️ Screen Navigator — Live screenshot interpretation and UI action execution (The UI Navigator category)
  • 📄 Document Intelligence — PDF and spreadsheet upload with structured data extraction
  • 🤝 Multi-agent ADK — ARIA as an orchestrator, delegating to specialized sub-agents via the Agent Development Kit
  • 📱 Mobile — Native camera integration for real-time visual Q&A on the go

🙏 Final Thoughts

ARIA started as an answer to a challenge and became something we genuinely want to use every day. The Gemini Live Agent Challenge pushed us to stop thinking about AI as a text box and start thinking about it as a presence — one that sees what you see, understands what you mean, and responds in the most natural way possible.

We're just getting started.

Built with ❤️ for the Gemini Live Agent Challenge · March 2026

Built With

  • 2.0
  • agent
  • ai
  • api
  • apis
  • architecture
  • base64
  • cloud
  • code
  • css3
  • development
  • devpost
  • encoding
  • firestore
  • flash
  • frameworks
  • gemini
  • gen
  • github
  • google
  • html5
  • image
  • javascript
  • kit
  • languages
  • libraries
  • live
  • models
  • multimodal
  • node.js
  • other
  • platform
  • platforms
  • react
  • real-time
  • rest
  • run
  • sdk
  • services
  • speech
  • speech-to-text
  • stateless
  • streaming
  • text-to-speech
  • tools
  • vertex
  • vision
  • vs
  • web
Share this project:

Updates

Submission history