AgriLive: Multimodal Farm Assistant

An AI farming companion that analyzes crops through live video and gives farmers instant voice guidance in the field.

Comment

🌱 AgriLive – Real-Time Multimodal AI Companion for Farmers

AgriLive is a real-time multimodal AI farming assistant that helps farmers diagnose crop diseases and receive expert guidance instantly using voice conversation and live camera analysis.

Inspiration

Farming today feels increasingly unpredictable. Weather patterns that farmers relied on for generations are shifting, and sudden storms, pests, and diseases can wipe out crops almost overnight.

This problem isn't limited to one region. From the American Midwest to farming communities in Malappuram, Kerala, climate volatility is making agriculture harder to manage every year.

When something goes wrong in the field, farmers don’t have time to navigate complex dashboards or search through technical documentation. What they need is instant guidance, right where they are working.

That idea sparked AgriLive.

Inspired by the Gemini Live Agent Challenge, we wanted to move beyond the traditional “type a question into a text box” approach and build something far more natural — an AI assistant farmers could simply talk to, just like speaking with an agricultural expert standing beside them.

What It Does

AgriLive functions like an empathetic agricultural extension officer in your pocket.

Farmers can start a live conversation with AgriBot using voice, ask questions about their crops, and receive guidance instantly.

The system dynamically retrieves real-time weather information and agricultural advisories using Google Search Grounding, allowing the AI to provide context-aware recommendations tailored to the farmer's location.

But farming problems are often visual.

If a farmer notices a strange spot on a leaf or a wilting plant, they can simply point their phone camera at it.

While the voice conversation continues, a background Vision Agent analyzes the video frame and identifies possible plant diseases or stress conditions. Within seconds, the system returns:

  • A probable diagnosis
  • A confidence score
  • A recommended treatment plan, prioritizing organic and safe practices

Instead of searching forums or waiting for expert advice, the farmer gets immediate assistance directly in the field.

Key Features

  • 🎙 Real-time voice conversation with an AI agricultural assistant
  • 📷 Live camera crop diagnosis using multimodal AI vision analysis
  • 🌦 Weather-aware farming advice powered by search grounding
  • 🌱 Organic treatment recommendations prioritizing safe practices
  • ⚡ Low-latency multimodal interaction using streaming audio

How We Built It

AgriLive was designed with a cloud-native architecture focused on low latency and real-time interaction.

🧠 The Brain

For voice conversations we used:

  • gemini-live-2.5-flash-native-audio
  • WebSocket-based bidirectional streaming

This enables natural real-time dialogue between farmers and the AI assistant.

Farmers speak normally, and the AI responds almost instantly.

👁 The Eyes

For visual analysis we implemented a separate REST API pipeline powered by:

  • gemini-2.5-flash

To guarantee reliable outputs for the UI, we enforced Pydantic Structured Outputs, ensuring the model always returns valid JSON responses.

Mathematically, responses follow a structured schema:

[ Response = {diagnosis,\ confidence,\ treatment_plan} ]

This guarantees the frontend receives predictable structured data every time.

☁️ Infrastructure

The backend runs on a FastAPI server deployed to Google Cloud Run.

To support continuous AI interaction we configured:

  • Always-On CPU
  • 2GB memory allocation

This allows the server to manage simultaneous audio streaming and visual analysis efficiently.

Tech Stack

AI Models

  • Gemini Live API
  • Gemini Vision (Gemini 2.5 Flash)

Backend

  • FastAPI
  • WebSocket Audio Streaming

Infrastructure

  • Google Cloud Run

Data Handling

  • Pydantic Structured Outputs

Frontend

  • Real-time camera interface
  • Voice interaction UI

Challenges We Encountered

🌍 Network Latency

Our backend ran in Iowa (us-central1), while testing occurred in Kerala, India.

Streaming raw 16-bit PCM audio across continents introduced noticeable latency and playback stuttering.

To solve this, we implemented a custom frontend Jitter Buffer, which temporarily stores incoming audio packets before playback to smooth the stream.

If incoming packets are represented as:

[ A = {a_1, a_2, a_3, ... , a_n} ]

The buffer produces smoother playback:

[ Playback = f(A_{buffer}) ]

This significantly improved audio stability.

🔌 WebSocket Session Drops

Another issue occurred when the user stayed silent.

The API interpreted silence as a disconnected session and closed the WebSocket.

Our solution was a simple but effective workaround.

When the AI speaks, the frontend streams zero-value audio samples (silence) back to the server, preventing the connection from closing.

This created a stable "Walkie-Talkie Mode" that keeps the session alive.

🎭 Making the AI Sound Human

Technical capability alone isn't enough. Farmers interacting with the system may be stressed about potential crop loss.

We carefully refined system prompts to ensure the assistant:

  • responds calmly and clearly
  • acknowledges farmer concerns
  • avoids recommending banned pesticides
  • prioritizes safer organic treatments

Achieving the right tone required extensive prompt tuning.

Lessons Learned

Building AgriLive turned out to be very different from building traditional web applications.

1️⃣ Real-time systems are complex

Managing stateful WebSockets, streaming audio, and handling different sample rates pushed us beyond typical REST-based application design.

2️⃣ Browser audio processing is surprisingly challenging

Handling raw PCM streams, buffering packets, and performing real-time downsampling inside the browser required significant experimentation.

3️⃣ Decoupling AI agents improves reliability

Separating responsibilities between agents proved extremely effective:

  • Live Agent handles human conversation and empathy
  • Vision Agent handles structured crop analysis

This separation creates a system that feels seamless to the user while remaining stable under the hood.

Final Thoughts

AgriLive is still evolving, but the goal remains simple:

Technology should meet farmers where they are — directly in the field.

By combining voice conversation, visual crop diagnosis, and real-time agricultural data, AgriLive transforms a smartphone into a practical farming companion.

Seeing a farmer simply point their camera at a leaf and instantly receive meaningful guidance reminds us that the best technology doesn’t feel complicated — it just feels helpful.

Built With

Share this project:

Updates

posted an update —

V1.0 Submitted: Engineering AgriLive for the Real World!

Hey everyone! I just officially hit "Submit" on AgriLive: Multimodal Farm Assistant for the Gemini Live Agent Challenge, and I wanted to share a quick look under the hood at how this project evolved from a raw concept into a production-ready application.

When I started, the goal was simple: help farmers in regions like Kerala fight climate volatility without forcing them to navigate complex text menus. But building a real-time, voice-and-vision AI over intercontinental networks? That required some serious engineering pivots.

The Evolution & Key Features

  • From Text to Native Audio: We completely ditched the text-box paradigm. AgriLive now uses gemini-live-2.5-flash-native-audio over WebSockets for a continuous, bidirectional, and empathetic voice conversation.
  • The "Walkie-Talkie" Protocol: Vertex AI expects a continuous audio stream and will drop the connection if the user goes silent. My favorite hack of the weekend? Building a "Walkie-Talkie" mode that dynamically streams an array of zeros (pure silence) to Google's servers whenever the AI is speaking, keeping the WebSocket completely stable!
  • Beating Ocean Latency: Streaming raw audio packets from India to the us-central1 servers caused brutal audio stuttering. I engineered a custom frontend Jitter Buffer with a "fast-track re-entry" mechanism to ensure the 24kHz PCM audio playback stays incredibly smooth, even on weak rural networks.
  • Concurrent Vision Agent: While the user is talking, they can snap a picture of a diseased crop. A backend cascading fallback engine routes the image to gemini-2.5-flash, enforcing a strict Pydantic Structured Output to guarantee the UI gets a perfectly parsed JSON diagnosis every single time.

Snippet Spotlight: The Fast-Track Jitter Buffer

Here is a peek at the logic that stitches delayed audio packets back together mid-sentence without forcing the browser to wait for a full buffer stockpile:

// Push incoming Float32 PCM data to the jitter buffer
audioQueue.push(float32);

// Fast-track re-entry: If the network lags but the AI is already mid-sentence,
// bypass the buffer threshold and play the audio immediately!
const isMidSentence = btnStart.classList.contains("speaking");

if (!isPlaybackStarted && (audioQueue.length >= JITTER_BUFFER_THRESHOLD || isMidSentence)) {
    startPlaybackLoop();
}

Log in or sign up for Devpost to join the conversation.

Submission history