Kisan Mitra

Empowering rural farmers with crop guidance - just speak, show, or type your question and Get answer

Comment

Kisan Mitra - Project Story

Inspiration

India is home to over 100 million farming households, with a significant portion cultivating oilseed crops like groundnut, sunflower, soybean, and sesame. These farmers often face critical challenges:

  • Limited access to experts: Agricultural extension officers are scarce in rural areas, with ratios as high as 1:20000(officer to farmers)
  • Language barriers: Most farmers speak regional languages (Hindi, Telugu, Kannada, etc.) while technical resources are often in English
  • Time-sensitive decisions: Crop diseases can devastate yields within days if not identified and treated promptly
  • Digital divide: Many farmers are more comfortable speaking than typing on smartphones

We were inspired by a simple question: What if every farmer could have a knowledgeable agricultural expert in their pocket, available 24/7, who speaks their language and can even "see" their crops?

When Google announced the Gemini Live API with native audio capabilities, we saw the perfect opportunity to build this vision into reality.

What We Learned

Technical Learnings

  1. Gemini Live API Architecture: We learned how bidirectional streaming works with WebSockets, handling real-time audio at 16kHz input and 24kHz output, and managing turn-taking in conversations.

  2. Agent Development Kit (ADK): We discovered the power of ADK's AgentTool pattern, which allows a root agent to orchestrate multiple specialized sub-agents while maintaining conversational context.

  3. Multi-Agent Orchestration: Building a system where:

    • The Root Agent (Gemini Live) handles real-time voice/video
    • The Knowledge Agent (Gemini Flash) queries the crop database
    • The Vision Agent (Gemini Flash) analyzes images for disease detection

  4. Knowledge Grounding: We learned the importance of grounding AI responses in authoritative data sources to prevent hallucinations - critical when farmers' livelihoods depend on accurate advice.

Domain Learnings

  • Oilseed crops have vastly different management practices across India's diverse agro-climatic zones
  • Disease identification requires understanding both visual symptoms AND contextual factors (season, region, crop stage)
  • Farmers prefer conversational, actionable advice over technical jargon

How We Built It

Architecture

┌─────────────────┐     WebSocket      ┌─────────────────────────────────┐
│   Mobile/Web    │◄──────────────────►│      FastAPI Backend            │
│   Frontend      │   Audio/Text/Image │      (Cloud Run)                │
└─────────────────┘                    └───────────────┬─────────────────┘
                                                       │
                                                       ▼
                                       ┌───────────────────────────────┐
                                       │   ADK Runner                  │
                                       │   ┌─────────────────────────┐ │
                                       │   │ Root Agent (Live Audio) │ │
                                       │   │ gemini-live-2.5-flash   │ │
                                       │   └──────────┬──────────────┘ │
                                       │              │                │
                                       │   ┌─────────┴─────────┐      │
                                       │   ▼                   ▼      │
                                       │ ┌─────────┐   ┌───────────┐  │
                                       │ │Knowledge│   │  Vision   │  │
                                       │ │ Agent   │   │  Agent    │  │
                                       │ └────┬────┘   └─────┬─────┘  │
                                       └──────┼──────────────┼────────┘
                                              ▼              ▼
                                       ┌──────────┐  ┌─────────────┐
                                       │Crop KB   │  │Disease KB   │
                                       │(10 crops)│  │(+ images)   │
                                       └──────────┘  └─────────────┘

Development Process

Phase 1: Knowledge Base Integration

  • Partnered with ICAR-IIOR to obtain authoritative crop management data
  • Processed 10 oilseed crop documents into structured text files
  • Built a fuzzy-matching system to handle variations in crop names (e.g., "groundnut" vs "peanut" vs "moongphali")

Phase 2: Multi-Agent System

  • Designed the agent hierarchy using ADK's AgentTool pattern
  • Implemented the Knowledge Agent with infer_crop() and read_crop_files() tools
  • Built the Vision Agent with disease detection capabilities and reference image matching

Phase 3: Real-Time Interface

  • Created a mobile-first web UI with three interaction modes:
    • Text Chat: For low-bandwidth scenarios (text-only responses)
    • Voice Input: Full duplex audio with Gemini Live
    • Video Query: Camera streaming for visual crop analysis

Phase 4: Deployment & Optimization

  • Deployed on Google Cloud Run with automated scaling
  • Implemented retry logic for API rate limiting
  • Added WhatsApp integration for broader accessibility

Tech Stack

Layer Technology
AI Framework Google ADK (Agent Development Kit)
Models Gemini Live 2.5 Flash (audio), Gemini 2.5 Flash (text/vision)
Backend FastAPI, Python 3.12
Real-time WebSockets (bidirectional streaming)
Hosting Google Cloud Run
AI Platform Vertex AI
Frontend Vanilla HTML/CSS/JS (mobile-first)

Challenges We Faced

1. Audio Latency & Synchronization

Challenge: Ensuring smooth real-time audio playback without gaps or overlaps.

Solution: Implemented a scheduled audio buffer system that queues PCM chunks and plays them at precise intervals using the Web Audio API's currentTime scheduling.

2. Multi-Agent Context Sharing

Challenge: The root agent needed to combine responses from both Knowledge and Vision agents coherently.

Solution: Used ADK's AgentTool with skip_summarization=True to pass full responses back to the root agent, allowing it to synthesize comprehensive answers.

3. Rate Limiting (429 Errors)

Challenge: During testing, we frequently hit Gemini API rate limits.

Solution: Implemented exponential backoff retry logic in the agent configurations:

generate_content_config=types.GenerateContentConfig(
    http_options=types.HttpOptions(
        retry_options=types.HttpRetryOptions(
            initial_delay=2,
            attempts=5,
        ),
    ),
)

4. Text vs Audio Mode Separation

Challenge: Users wanted text-only chat without audio responses for low-bandwidth scenarios.

Solution: Created separate WebSocket endpoints:

  • /ws/{user_id}/{session_id} - Audio mode (voice/video)
  • /ws/text/{user_id}/{session_id} - Text-only mode

5. Multilingual Support

Challenge: Farmers speak diverse languages (Hindi, Telugu, Tamil, etc.).

Solution: Leveraged Gemini's native multilingual capabilities with a system prompt that instructs the agent to detect and respond in the user's language.

What's Next

  • Expand crop coverage: Add more crops beyond oilseeds
  • Offline mode: Cache common queries for areas with poor connectivity
  • WhatsApp bot: Full integration for farmers without smartphones
  • Regional voice models: Fine-tune for Indian language accents
  • Farmer community: Enable farmers to share successful practices

Built with love for Indian farmers by Sriman Katipally and Nimesh CH

Built With

Share this project:

Updates

Submission history