Agro-doc

AI crop doctor for farmers snap a photo of a sick leaf, get an instant disease diagnosis from a fine-tuned ML model, plus a treatment plan in your own language.

Comment

Inspiration

Smallholder farmers lose 20–40% of their annual crop yield to plant diseases and pests that go undiagnosed until it's too late. The real bottleneck isn't a lack of solutions — it's access. Certified agronomists are concentrated in cities, extension services are stretched thin, and a single farm visit can cost more than a week's income. By the time a farmer reaches an expert, the infection has already spread. We wanted to put an expert in every farmer's pocket — and crucially, one that speaks their language.

What it does

AgroDoc lets a farmer photograph a diseased crop leaf and instantly get:

  • A diagnosis — a fine-tuned MobileNetV2 model identifies the disease from 38 classes across 14 crops, showing the top-3 predictions with confidence scores.
  • A treatment plan — an LLM generates a structured, plain-language plan: the root cause, step-by-step treatment, organic alternatives, prevention tips, and an urgency level (low / medium / high).
  • Answers — a contextual chat lets the farmer ask any follow-up question grounded in their specific diagnosis.
  • In their language — every piece of advice translates instantly into Hindi, Spanish, Swahili, or French with one click.

The whole experience is a simple 4-step flow: Upload → Diagnosis → Treatment → Ask AgroDoc.

How we built it

Machine Learning: We fine-tuned a MobileNetV2 model (pre-trained on plant-disease data) on a balanced subset of the PlantVillage dataset — 6,650 training and 1,140 validation images across all 38 classes — using PyTorch and HuggingFace Transformers. We used best-checkpoint selection, saving the model whenever validation accuracy improved, which gave us 87.7% validation accuracy and a 0.872 macro F1.

Backend: A FastAPI server exposes four endpoints — /predict (image classification), /advice (treatment plan generation), /translate (batch translation), and /chat (contextual Q&A). The LLM features are powered by Featherless.ai using Qwen2.5-7B-Instruct.

Frontend: A React + Vite single-page app styled with Tailwind CSS and animated with Framer Motion. It includes a live "Model Performance" panel showing our real training metrics, a step-progress indicator, and a mobile-first design with in-field camera capture.

Challenges we ran into

  • A CORS error that was really a 500. Our /translate endpoint kept failing with a CORS error in the browser. The real cause was an unhandled backend exception — when FastAPI crashed, the error response bypassed the CORS middleware, so the browser only saw a missing CORS header. We fixed it with broader exception handling and a global error handler that always returns proper JSON.
  • Model overfitting. Validation accuracy peaked at epoch 2 (87.7%) then dropped at epoch 3. We added best-checkpoint saving so the model keeps the best-performing weights, not just the final ones.
  • Slow translation. Translating the whole treatment plan was sluggish. We switched from five separate API calls to a single batched call and added client-side caching, so re-selecting a language is instant.
  • A broken layout. Leftover Vite default CSS was pinning all content to the left of the screen. Tracking it down to the #root and body styles in index.css fixed the centering.

Accomplishments that we're proud of

  • A real, fine-tuned ML model — not just an API wrapper — with honest, reproducible metrics (87.7% accuracy, full confusion matrix and per-class F1 scores) shown live in the app.
  • Genuine accessibility — multilingual support that puts expert advice in the farmer's mother tongue, which is the real barrier for our target users.
  • A polished, production-feeling UI with a transparent "Model Performance" panel, so the model isn't a black box — judges and farmers alike can see exactly how well it performs.
  • An end-to-end working product built from scratch: data prep, training, backend, frontend, and documentation.

What we learned

  • How to fine-tune a vision transformer/CNN and read training dynamics — recognizing overfitting from the validation curve and responding with best-checkpoint selection.
  • A subtle but important debugging lesson: a CORS error in the browser can actually be a hidden server-side 500, because crashed responses skip the CORS middleware.
  • How to design AI UX that builds trust — showing confidence scores, model metrics, and honest "low confidence" warnings instead of pretending the model is always right.
  • Practical LLM integration: structured JSON prompting, batching calls for speed, and caching results client-side.

What's next for AgroDoc

  • Offline / edge deployment — export the model to TFLite or ONNX for on-device inference in regions with no connectivity.
  • More crops and pests — expand beyond 14 crops to cassava, rice, and legumes critical to Sub-Saharan Africa and South Asia, and add insect-pest identification.
  • Voice interface — speech-to-text and text-to-speech for farmers who aren't comfortable typing.
  • Severity scoring — estimate how far an infection has spread from the image.
  • WhatsApp / SMS channel — meet farmers on platforms they already use.
  • Outbreak mapping — aggregate anonymized diagnoses to surface regional disease patterns in real time.

Built With

Share this project:

Updates