FarmSense: AI Agronomist for Smallholder Farmers

Inspiration

Smallholder farmers produce roughly a third of the world's food, yet they have almost no access to expert agronomic advice when a crop fails. Extension workers are stretched thin, AI landscape tools only show aesthetics, and generic online manuals can't account for the rainfall this week near this village. We wanted to build a tool that answers the question every farmer actually asks: "My maize leaves are curling — what's happening and what do I do right now?" — grounded in real-time, hyper-local data, not generic templates.

What It Does

FarmSense is a single intelligent agent that acts as a personal AI agronomist. A farmer describes their situation in plain language (crop, location, symptoms), and the agent automatically:

Normalizes the location to precise coordinates.
Queries 4 independent data streams in parallel — recent climate trends, nearby pest outbreaks, soil profiles, and semantic agronomic knowledge.
Synthesizes the data into a structured advisory with a risk level, root-cause diagnosis citing specific numbers (e.g., "rainfall 43% below average"), and step-by-step immediate and preventive actions.
Logs every advisory for audit and triggers automatic alerts when risk is critical.

All of this happens in under 60 seconds from a single chat message — accessible via Kibana Agent Chat or a Telegram bot on any smartphone.

How We Built It

Elasticsearch Serverless as the unified data platform — storing climate time-series, geo-located pest outbreaks, soil profiles, crop calendars, and agronomic text, each with purpose-built mappings (geo_point, date, semantic_text).
ES|QL for all geo-distance and time-window analytical queries (climate within 200 km / 90 days, pests within 300 km / 30 days, soils within 100 km).
ELSER v2 for semantic search over crop-knowledge documents — enabling symptom-to-diagnosis matching without exact keyword overlap.
Elastic Agent Builder to wire the LLM to 7 custom tools and orchestrate the 4-step pipeline (Intake → Intelligence → Advisory → Log).
Elastic Workflows for a zero-code automation layer that indexes advisories and fires webhook alerts for CRITICAL risk levels.
Python + uv for data ingestion scripts and agent configuration, keeping the backend lightweight and reproducible.
Telegram Bot (FastAPI + ngrok) as a mobile-first frontend — farmers interact via a familiar chat interface in real time.

Challenges We Ran Into

ES|QL + geo_point precision: Early queries silently returned zero results because ST_DISTANCE required exact WKT format (POINT(lon lat)) and the agent was passing coordinates in the wrong order. Debugging involved adding explicit format instructions to the agent prompt.
ELSER deployment timing: On Serverless, ELSER needed to be started and fully allocated before semantic queries would return results. Until then, the tool silently timed out. We solved this with graceful degradation — the agent still produces an advisory using the other three data sources.
Synthetic data realism: Getting synthetic rainfall patterns that actually trigger the agent's "drought stress" diagnosis (not just random numbers) required carefully modelling West African dry-season curves and regional baselines.
Agent instruction tuning: The LLM initially stopped after Step 1 (Intake) and returned partial results. We had to explicitly instruct it with "Do NOT stop after Step 1" and enforce the 4-step pipeline in the system prompt.

Accomplishments That We're Proud Of

End-to-end in one command: ./do_everything.sh creates indices, ingests data, deploys ELSER, sets up all tools, and registers the agent — ready to chat in under 2 minutes.
Real data grounding: Every advisory cites specific numbers from the queried data ("rainfall 43% below average", "Fall Armyworm outbreak 180 km away") — no hallucinated statistics.
5 validated demo scenarios covering maize/Nigeria drought + FAW, rice/Bangladesh flooding + blast, wheat/India aphids, cassava/Kenya nutrient deficiency, and tomato/Ethiopia late blight — each produces clinically accurate advisories.
Zero-code alerting: Critical pest risks automatically trigger webhook alerts via Elastic Workflows without any custom backend code.
Telegram frontend: Farmers can interact with the full AI pipeline from a phone via Telegram — no app download, no web browser required.

What We Learned

ES|QL is powerful for geo-analytical queries — once you get the syntax right, combining ST_DISTANCE, BUCKET, and STATS in a single pipe-separated query is far more readable and faster to iterate on than equivalent DSL.
ELSER makes semantic search trivial — mapping a field as semantic_text and letting the model handle embeddings removed an entire preprocessing pipeline we thought we'd need for symptom matching.
Agent prompt engineering is critical — the difference between a useful advisory and a hallucinated one came down to precise tool-use instructions in the system prompt, not model size.
Elastic Workflows replace custom glue code — what would have been a separate microservice for alert dispatch became a declarative YAML workflow attached to the agent in minutes.

What's Next for FarmSense

Live data integration: Connect to real APIs — NASA POWER for climate, FAO EMPRES for pest outbreaks, ISRIC SoilGrids for soil — replacing synthetic data with live feeds.
Multi-language support: Translate advisories into Yoruba, Hindi, Swahili, and Bangla so farmers can receive advice in their native language.
Image-based diagnosis: Accept crop photos via Telegram and use vision models to augment the text-based symptom analysis.
Feedback loop: Let farmers report "this advice worked / didn't work" to calibrate future advisories and build a regional knowledge base.
Offline mode: Cache recent advisories and common diagnoses for areas with intermittent connectivity.
WhatsApp integration: Expand beyond Telegram to WhatsApp, which has higher penetration in Sub-Saharan Africa and South Asia.

Agent Brief

FarmSense is an AI-powered agronomist agent built on Elastic Agent Builder that delivers free, instant, and hyper-local crop advisories to smallholder farmers and extension workers. It solves a critical problem: when a farmer's crop shows distress — yellowing leaves, curling, stunted growth — they currently have no way to get expert-level, data-grounded advice without waiting days for an extension visit or navigating complex institutional tools. FarmSense changes this by letting farmers describe their situation in plain language and receiving a structured, actionable advisory in under 60 seconds.

The agent is built as a single orchestrator (FarmSense Advisor) with 7 specialized tools. When a farmer sends a message, the agent first normalizes their location using ES|QL geo-queries against a crop-calendar index to resolve region names into precise coordinates. It then runs four parallel intelligence queries: geo-climate analysis (rainfall and temperature trends within 200 km over the last 90 days via geo_point and ST_DISTANCE), pest outbreak lookup (active outbreaks within 300 km using geo-distance filtering), soil profile matching (soil type, drainage, and water-holding capacity for the nearest profiles), and semantic crop knowledge search using ELSER v2 to match symptom descriptions against FAO/CGIAR agronomic guidelines without requiring exact keyword overlap. The agent synthesizes all four data streams into a structured advisory with a risk level, root-cause diagnosis citing specific data points, and concrete immediate and preventive actions. Finally, the advisory is logged via Elastic Workflows, which also autonomously trigger webhook alerts when the risk level is CRITICAL.

Two Elastic features stood out as particularly impactful. First, ES|QL made geo-spatial and time-series analytical queries remarkably clean — combining ST_DISTANCE, date filtering, and statistical aggregations in a single readable pipe chain replaced what would have been complex nested DSL. Second, ELSER v2 semantic search eliminated the need for a separate embedding pipeline; simply mapping a field as semantic_text allowed the agent to match "leaves are weird" to documents about "chlorosis" and "abnormal foliage" out of the box.

The biggest challenge was agent instruction tuning. The LLM initially stopped after resolving the location and returned incomplete results. Getting it to reliably execute all four intelligence queries and produce the full structured advisory required careful prompt engineering, including explicit "Do NOT stop after Step 1" instructions and a rigid output format template. This taught us that the orchestration layer — how the agent is told to use its tools — matters just as much as the tools themselves.