FarmWise

Smarter Farming, Stronger Harvests

Comment

Inspiration

India is an agrarian economy, yet smallholder farmers—especially in regions like Maharashtra—face devastating crop losses (up to 20-30%) due to delayed disease diagnosis and lack of timely expert advice. Traditional extension services are overstretched, and language barriers often prevent farmers from accessing modern agricultural knowledge. We wanted to build a bridge between cutting-edge AI and the field. FarmWise was inspired by the idea of putting a PhD-level Agronomist in every farmer's pocket—one that speaks their local language, understands local market dynamics, and now, thanks to the Live API, can actually "see" the field through the farmer's eyes.

What it does

FarmWise is a comprehensive "Super App" for Indian farmers with two distinct roles (Farmer & Consumer): -FarmWise Live: A real-time, multimodal video consultation tool. Farmers can open a live camera feed, zoom in on a pest or leaf, and talk naturally to the AI. The AI "sees" the video stream in real-time and provides instant verbal advice, just like a human expert standing next to them. -AI Crop Doctor: For high-precision diagnosis, farmers take a high-res photo. The app uses Gemini 3's multimodal vision and reasoning capabilities to identify the disease, explain symptoms, and provide a chemical and organic treatment plan. -FarmWise AI Assistant: A conversational assistant with three specific modes: -->Fast: For quick queries. -->Deep Think: Uses Gemini's thinkingConfig to reason through complex scenarios (e.g., "It rained heavily after sowing, what should I do?"). -->Store Locator: Uses Search Grounding to find real-world agricultural stores nearby. -Market Pulse & Weather: Provides real-time mandi prices and hyper-local weather forecasts with actionable farming alerts. -Farm-to-Table Store: A marketplace where farmers can list produce and consumers can buy fresh items directly, cutting out middlemen.

How we built it

We built FarmWise as a Progressive Web App (PWA) using React 18, TypeScript, and Tailwind CSS. The core intelligence is powered by the Google Gemini 3 API. -Real-Time Video Streaming (Live API): We integrated the gemini-2.5-flash-native-audio-preview model to create a continuous multimodal session. -->Visuals: We implemented a custom frame-capture engine that grabs video frames from the device's camera (supporting pinch-to-zoom and front/back switching), compresses them to optimized JPEGs, and streams them to the model via WebSockets. -->Audio: We built a raw PCM audio processor (16kHz input / 24kHz output) to handle bidirectional voice communication without heavy external libraries, ensuring low latency on rural 4G networks. -Multimodal Diagnosis: For the static "Crop Doctor," we used gemini-3-pro-preview. We enabled thinkingConfig: { thinkingBudget: 1024 }, which forces the model to internally debate symptoms (e.g., "Is this Early Blight or Bacterial Spot?") before outputting a diagnosis. -Structured Outputs: We utilized responseSchema to force Gemini to return strictly typed JSON, ensuring the UI remains robust even when the AI translates complex agricultural terms into languages like Marathi or Tamil. -Smart Chat: We implemented a mode-switcher. "Deep Think" utilizes the Pro model with a high thinking budget, while "Find Stores" utilizes the Flash model with Google Search grounding.

Challenges we ran into

-Streaming Video Optimization: Sending high-res video frames instantly saturated the bandwidth. We had to implement an adaptive canvas strategy that downscales frames and adjusts compression quality on the fly to maintain a smooth conversation flow. -Hallucinations in Local Languages: Initially, the model would invent Marathi words for technical diseases. We solved this by refining System Instructions to enforce a specific persona ('Puck') that speaks strictly in the requested language or English with an Indian accent. -Button Sizing & Layouts: Translating "Add to Cart" or "Instant Disease Check" into languages like Malayalam often broke the UI. We implemented responsive text sizing and whitespace constraints to handle this.

Accomplishments that we're proud of

-"Eyes on the Field": Successfully implementing the Live API with video streaming. Seeing the AI respond to visual cues in real-time ("I see yellowing on the leaf margins...") feels magical. -The "Thinking" UI: We are proud of integrating the gemini-3-pro reasoning capabilities. It doesn't just guess; it analyzes. -True Multilingual Support: The app fully supports 11 Indian languages. The entire interface, including the Live AI's spoken voice and persona, adapts instantly to the user's preferred language. -Zero-Latency Feel: Despite the heavy AI lifting, the app feels snappy due to optimistic UI updates and efficient state management.

What we learned

-Video is the Ultimate Context: Text descriptions of crop diseases fail often because farmers might not know the right terminology. Giving the AI "sight" via the Live API solves this context gap completely. -Schema is King: Strictly defining responseSchema is the difference between a cool demo and a production-ready app. -Localization is Hard: Translation is not just about swapping words; it's about cultural context. A "store" in the US is a "dukan" or "kendra" in India, and the AI needs to know that context to search effectively.

What's next for FarmWise

-Offline Capability: Integrating Gemini Nano to allow basic crop diagnosis directly on the device, even without an internet connection in remote fields. -Satellite Analysis: Integrating satellite imagery to provide macro-level advice on irrigation and harvest timing. -Community Board: A feature for farmers to share success stories and warn neighbors about pest outbreaks. -Real Payments: Integrating UPI for actual transactions in the consumer marketplace.

Built With

Share this project:

Updates