Cloud Run
Deployment Automation Trigger
Deployment Automation Files
Cloud Run Logs
Cloud Build

Prometheus - Multimodal AI Solar Advisor

Google Cloud / Gemini Live Agent Challenge - Category: Live Agents 🗣️

Inspiration

We have always been curious about solar energy, but every time we looked into it the process felt like a black box - roof suitability, system sizing, incentives, installer quotes - each step buried behind a different website, phone call, or form. It never felt accessible.

That curiosity turned into a concrete idea during a trip to Miami. While exploring the area, we came across a solar installation set up on one of the lakes - floating panels generating clean energy from open water. That image stuck with us. Solar wasn't just for rooftops. It could be on patios, in backyards, on water. And yet most people don't even know where to start with a standard rooftop installation, let alone something more creative.

That moment crystallised the question: what if opening that black box was as simple as having a conversation? Not filling out forms, not calling three different companies, not decoding a tax credit PDF - just talking to an advisor who already knows your house, your bill, your roof, and your backyard. That's what we set out to build with Prometheus (the name being inspired from Greek mythology wherein Prometheus stole fire from the gods and gave it to ordinary people)

The problem is real. The US residential solar market is valued at $7.45 billion and growing at 14.4% annually - yet the residential segment saw a 32% decline in installations in 2024 (SEIA), driven largely by consumer confusion, incentive blindness, and process friction. A striking number of homeowners are unaware of the federal 30% Investment Tax Credit (ITC) - worth an average of $8,000–$10,000 - simply because nobody has assembled that information in one accessible place. Meanwhile, Google has pledged over $20 billion toward solar infrastructure. We wanted to bring that same level of solar intelligence to the individual homeowner.

The mission: make solar accessible to every homeowner - not just the ones with the time, money, or knowledge to navigate the process alone.

What It Does

Prometheus is a fully agentic, multimodal AI solar advisor that guides homeowners through the entire solar evaluation journey in a single, real-time voice conversation. Unlike traditional chatbots that require strict turn-taking, Prometheus supports active barge-in (demostrated in our demo): homeowners can interrupt at any point to ask follow-up questions or clarify details. The agent immediately stops, listens, and pivots - behaving like a natural conversation partner, not a static interface.

Prometheus Sees (camera, photos, PDFs), Hears (live voice), and Speaks (Gemini Live native audio) through a seamless six-step session:

A complete session looks like this:

Upload your electricity bill - the agent extracts your consumption, billing period, and address from the PDF using Google Document AI OCR, so it knows your exact energy usage before any analysis begins.
Speak your address - the agent calls the Google Solar API to retrieve LiDAR-derived data for that specific property: roof geometry, pitch, azimuth, usable area, and annual solar irradiance. It maps your bill to the right panel configuration and delivers a full financial breakdown - system cost, panel count, payback period, and annual savings.
Discover your incentives - the agent automatically applies the federal 30% ITC and performs a live web search via the Google Custom Search API for current state rebates and utility programs. Most homeowners have never seen this assembled in one place.
Point your camera at any outdoor space - backyard, patio, courtyard, carport, or even a water feature. The agent captures the frame and uses Gemini 2.0 Flash computer vision to assess the space for solar potential, estimating usable area, panel count, and whether a canopy or ground-mount is more appropriate.
See photorealistic mockups on your actual property - using Gemini 3.1 Flash Image Preview, the agent edits your real Street View photo (rooftop) or your actual camera frame (outdoor space) to show exactly what a solar installation would look like - on your house, in your yard, at your scale. Not a generic stock render.
Find local installers and send RFPs - the agent uses the Google Places API to find real, rated solar installers near the property, then drafts personalised Request for Proposal emails with the homeowner's actual solar data and sends them via the Gmail API. The homeowner goes from "interested" to "three competitive quotes requested" in a single session.

Throughout the entire session, the user speaks naturally and can interrupt the agent at any point. Session memory persists key facts to Google Cloud Storage so the agent never asks the same question twice.

Demo note: Installer emails are sent to our own test accounts - we cannot send unsolicited commercial emails to real businesses during testing. The Gmail integration and RFP generation are fully functional; only the recipient addresses would change for production.

How We Built It

Multi-Agent Architecture

Prometheus is genuinely multi-agent - not a single model with a voice wrapper. Six specialised models each handle the task they are best suited for:

Role	Model	Modality
Voice conversation & orchestration	`gemini-live-2.5-flash-native-audio`	Audio in/out
Reasoning & tool decisions	`gemini-3.1-flash-lite-preview`	Text
PDF / document analysis	`gemini-3.1-pro-preview`	Text + document
Solar mockup image editing	`gemini-3.1-flash-image-preview`	Image in/out
Outdoor space computer vision	`gemini-2.0-flash-001`	Image → text
Image generation fallback	`imagen-3.0-generate-001`	Text → image

The Google Agent Development Kit (ADK) orchestrates all agents, tool callbacks, and session state. The server is a FastAPI WebSocket application bridging the browser to the ADK Live runner.

Barge-In / Interruption Handling

Prometheus implements interruption at every layer: the browser sends activity_start when the user begins speaking, the server immediately suppresses in-progress audio, a user_spoke loop guard prevents the model from responding to its own tool-generated speech, and activity_end re-enables the model when the user finishes. The agent behaves like a natural conversation partner, not a turn-based chatbot.

Calm, Familiar Persona will Greek Mythology Parallels

Prometheus speaks with the Chirp 3 HD Aoede voice - the same voice Google uses across its own Gemini products. This was a deliberate choice. Solar is a significant financial decision and that can be nerve-wracking - we wanted the agent to feel calm and reassuring from the first word, not clinical or transactional. The name carries the same intent: in Greek mythology, Prometheus stole fire from the gods and gave it to ordinary people. That is exactly what this agent does: taking solar intelligence that has only been accessible to experts and putting it into the hands of any homeowner with a phone and an electricity bill.

Composite Tool Pattern

In Gemini Live native audio, the model generates speech after every tool result - meaning a 4-step tool chain would produce 4 separate audio responses within one turn. We solved this by building composite tools (run_solar_analysis, calculate_combined_solar, send_all_rfps) that batch multiple internal operations and return a single result, so the model speaks exactly once per logical action.

Image Side-Channel Pattern

Returning a 2–3 MB image as base64 inside a tool response would overflow the voice model's 32K context window. Instead, the mockup tool stores image bytes in a server-side dict keyed by UUID and returns only the UUID to the voice model. The server drains the dict after each ADK event and pushes images directly to the browser - the bytes never enter the model's context.

Grounding - All Real Data

Every figure shown to the homeowner comes from a live API against their actual address and bill:

Data	Source
Roof geometry, panel count, irradiance	Google Solar API (LiDAR-derived)
Energy consumption, billing period	Google Document AI OCR from uploaded PDF
Property photo for mockup	Google Maps Street View Static API
Installer names, ratings, contact	Google Places API live query
State incentives & rebates	Google Custom Search live results
Federal ITC (30%)	Calculated from real Solar API system cost

Automated Cloud Infrastructure

Deployment is fully automated - Terraform provisions all GCP resources (Cloud Run, GCS, IAM, Artifact Registry, Secret Manager) as infrastructure-as-code, and Google Cloud Build deploys a new revision on every git push origin main. No manual steps after initial setup.

Technologies Used

Gemini Live · Gemini 3.1 Flash Lite Preview · Gemini 3.1 Pro Preview · Gemini 3.1 Flash Image Preview · Gemini 2.0 Flash · Imagen 3 · Google ADK · Google Document AI · Google Solar API · Google Maps Street View · Google Places API · Google Custom Search API · Gmail API · Google Cloud Run · Cloud Build · Secret Manager · Cloud Storage · Artifact Registry · Terraform · FastAPI

Challenges We Ran Into

Cracking the Solar API Cost Scenario Structure The Solar API doesn't return a single cost for a given bill - it returns a list of financial scenarios indexed by panel count. Finding the right configuration for a homeowner's specific monthly bill required iterating over all scenarios, computing which one's annual output best offsets the homeowner's actual consumption, and then extracting system cost and payback from that match. Getting this right - especially for edge cases where consumption falls between two scenarios - was one of the most demanding engineering problems in the project.

Voice Mode Behaves Completely Differently From Text Mode Everything that worked cleanly in text-based testing broke in voice mode. The model would speak 4–5 times within a single turn when tools were chained, often repeating financial summaries mid-flow. The solution - composite tools that batch multi-step operations into one call - required rethinking the entire tool layer and became the most important architectural pattern in the project.

Pre-Recorded Voice Announcements (Abandoned) To handle 15–30 second tool latency, we pre-baked seven WAV announcement files using Google Cloud TTS with the same Chirp 3 HD Aoede voice. In testing, they created an awkward double-voice effect with Gemini Live's real-time audio. We abandoned them in favour of injecting announcement text into the live model's context and letting it speak naturally. Pre-announcement of latency remains a known limitation - the agent can fall silent for up to 20 seconds during image generation.

Vertex AI Global vs. Regional Endpoints Gemini 3.1 preview models only exist on Vertex AI's global endpoint - calling us-central1 returns 404. Imagen 3 is regional only. This required a dual-client architecture with separate endpoint configurations, and was a significant debugging session before the root cause became clear.

Cloud Build Silently Wiping Environment Variables Every gcloud run deploy --set-env-vars call replaces all existing Cloud Run environment variables. Terraform correctly set MEMORY_BUCKET at provision time, but every CI/CD push from Cloud Build dropped it - silently, with no error. Session memory would fail after every deployment. The fix was adding the variable explicitly to cloudbuild.yaml.

Accomplishments That We're Proud Of

True Multimodality - Text, Image, Document, and Live Video in One Agent Prometheus processes every major input modality within a single session: spoken voice, PDF documents, uploaded photos, and live camera frames - while responding with real-time audio, photorealistic images, and structured emails. Each modality is handled by a different specialised model, not shoehorned into one.

Every Data Point Is Real - No Mock Data Anywhere Every figure the agent presents is grounded in a live API call against the homeowner's actual address and bill. There is no zip-code average, no stock photo, and no hardcoded example figure anywhere in the pipeline.

Barge-In That Actually Works Conversational interruption in voice agents is notoriously hard to get right. Prometheus handles it at every layer - browser, server, and model - so users can interrupt, redirect, or correct mid-sentence and the agent responds immediately and naturally.

Low Latency Through Multi-Agent Architecture By routing each task to the smallest capable model - lightweight reasoning for orchestration, heavier models only for documents and images - the conversational loop stays fast and responsive while still accessing deep capability when needed.

Photorealistic Mockups on the User's Actual Property Panels overlaid on the real Street View photo of their house, or on the photo they just took of their own backyard. Not a generic render - their property, their scale, their environment. This is the feature that turns interest into action.

What We Learned

The composite tool pattern is the key to usable voice agents. The single most important lesson: in Gemini Live native audio, the model speaks after every tool result. Designing tools around this constraint - batching logical workflows into composite calls - is not an optimisation, it's a prerequisite for a coherent experience.

Images and voice models need a side channel. You cannot pass binary data through a voice model's context. Storing images outside the model and passing only identifiers is a pattern that will apply to any multi-modal voice agent, not just this project.

Grounding must be architectural, not instructional. Telling an LLM "don't make up figures" is insufficient. The only reliable way to prevent hallucinated financial estimates is to ensure the model cannot produce a number without first calling a tool that retrieves real data. Session memory explicitly instructs the model to never use cached figures for new calculations - always call the relevant tool.

Native audio mode and text mode are fundamentally different runtime environments. Behaviour that works correctly in text testing can fail completely in voice mode. Testing in the actual modality, early and continuously, is non-negotiable.

Automated deployment pays for itself immediately. Having Terraform + Cloud Build wired up from day one meant every fix was live within minutes of a push. The one time it broke (the env var issue) was painful precisely because we had come to rely on it completely.

What's Next for Project Prometheus

Per-User Memory & Multi-User Support The current session memory uses a shared singleton - sufficient for demos. Production would key memory by authenticated user ID so each homeowner has their own persistent profile across sessions and devices. The GCS infrastructure is already in place; this is a one-line change.

Scaling to Other Renewable Energies The same voice-first, multimodal, tool-orchestrated architecture applies directly to wind micro-assessment, geothermal evaluation, battery storage sizing, and EV charging integration. Prometheus is designed as an energy advisor, not a solar-only tool.

Real Installer Connections Replacing our test email accounts with live installer directories and enabling real RFP dispatch - closing the loop from conversation to commercial transaction for actual homeowners.

Financing Guidance Integrating loan, lease, and PPA comparison to help homeowners understand not just the cost but the right way to finance a system given their tax position and credit profile.

Social Media Content: https://youtube.com/shorts/jpg5KlkKmzY?feature=share

Proof of Cloud Deployment: https://youtu.be/zOKj5VFdsfg