T.O.U.T. – Targeted Observation and Understanding Tool

A smart agricultural assistant that uses targeted image analysis and intelligent understanding to help farmers diagnose and manage crop diseases effectively.

Comment

T.O.U.T. – Targeted Observation and Understanding Tool for farmers

🚀 Inspiration

As a hobby farmer, I often encountered diseases on my crops and had no idea what I was looking at. I would spend hours searching online or asking around for answers—usually too late to save the plant. That made me realize: what if every farmer could instantly identify diseases just by snapping a photo? With my background in machine learning, my team and I decided to build T.O.U.T., an AI-powered assistant that could democratize crop disease diagnosis and treatment advice—especially for underserved farmers with limited access to experts.

🤖 What It Does

T.O.U.T. is a smart agricultural assistant that helps farmers:

  • Upload an image of a diseased plant leaf
  • Detect the crop and disease using an ensemble of MobileNetV3 models
  • Receive top-5 predictions with confidence scores
  • Chat with an intelligent bot that gives treatment advice, symptoms, and causes
  • (Experimental, Not live yet) Upload personal data to fine-tune the model via LoRA for more personalized detection.

🛠️ How We Built It

🌐 Web Interface & API Built with FastAPI and Jinja2, the web app allows users to:

  • Upload crop images
  • View the top-5 disease predictions with confidence scores
  • Chat with a local knowledge bot for treatment guidance

🖼️ Image Validation with CLIP We used OpenAI's CLIP ViT-B/32 to ensure uploaded images contain plant leaves. The validation logic checks with a formula that says: The image is considered valid if the plant score is greater than the non-plant score plus 0.20, and the plant score is greater than 0.60. If this condition fails, the user is notified to upload a better image.

🔬 Disease Classification An ensemble of MobileNetV3-Large-100 models was trained on a labeled dataset of 22 diseases across four crops: cashew, cassava, maize, and tomato. The model returns the top-5 predictions with confidence percentages.

🤖 Local RAG Knowledge Bot The chatbot combines:

  • MiniLM-L6-v2 sentence embeddings
  • FAISS for fast semantic vector search
  • Rule-based keyword matching for precise responses

The bot provides insights on:

  • Disease causes
  • Symptoms and effects
  • Diagnosis tips
  • Chemical treatment recommendations

🧠 Personalized LoRA Tuning (Experimental) We built a pipeline that allows farmers to upload their own farm data (images and labels), which is used to fine-tune a LoRA adapter. This aims to personalize disease detection based on local environmental conditions and crop variations.

Challenges We Ran Into

  • Image validation errors: Greenish non-leaf objects sometimes passed CLIP validation. This required refining prompt engineering and adjusting threshold logic.
  • Model generalization: Initial models performed poorly on real-world images with varied lighting and noise. We improved this with extensive data augmentation.
  • Chatbot reliability: Preventing hallucinations from the chatbot was a challenge. We solved this using hybrid RAG + rule-based logic.
  • LoRA personalization complexity: Creating a robust upload-and-tune system for each farmer was difficult. Managing dynamic weight storage, on-the-fly model updates, and compute limitations, as a result, this feature is not live in this demo yet.

Accomplishments That We're Proud Of

  • Developed a full-stack AI-powered farming assistant from scratch.
  • Successfully integrated image classification, semantic search, and local response generation.
  • Made the system CPU-only, so it runs even on low-power machines.
  • The idea of personalized LoRA fine-tuning.

What We Learned

  • How to apply deep learning to solve practical agricultural problems.
  • Best practices in CLIP prompt engineering** and image validation.
  • The power of ensemble models** for improving classification confidence.
  • Challenges of implementing real-time, per-user LoRA tuning with limited compute.

🔮 What's Next for T.O.U.T.

  • 🌍 Multilingual Support: To make the tool accessible across language barriers.
  • 📱 Mobile App Deployment: For farmers to use it in the field directly.
  • 🧠 Improved LoRA Fine-Tuning: For better user-specific performance and customization.
  • 🔎 Expanded Dataset: More diseases, more crops, more precision.
  • 📊 Analytics Dashboard: For agricultural officers and policy-makers to track disease outbreaks.

Built With

Share this project:

Updates