CropDoc

Point your phone at a crop leaf and get an instant diagnosis, targeted treatment and the right dose, fully offline. Farmers spray less and spray the right thing.

Comment

Inspiration

A farmer who sees brown spots on a leaf usually can't tell which disease caused them. Is it early blight, late blight, or a bacterial spot? The safe-looking answer is to spray a broad-spectrum chemical over the whole field. That costs money the farmer doesn't have, damages soil and runs off into water. Half the time it doesn't even target the real problem.

Most plant-disease apps assume a good data connection. Many of the fields that need them most have none. We wanted a tool that works in the field, with zero signal. It should name the specific disease, give a targeted treatment and work out how much to mix, so farmers spray less and spray the right thing.

What it does

  • 📸 Instant offline diagnosis: point the camera at a leaf and an on-device model identifies 38 conditions across 14 crops, with a confidence score and treatment steps.
  • 📈 Plot tracking: tag scans by field or row and see whether a plot is improving or getting worse.
  • 🧪 Dosage calculator: enter a plot size or plant count to get how much product and water to mix. On-device OCR reads a pesticide label and checks its rate against the typical range.
  • 🤖 Ask: an offline AI assistant that runs on the phone, grounded in CropDoc's treatment data, with voice input and read-aloud answers.
  • 🌍 11 languages, including Hindi, Swahili, Bengali and Urdu (right-to-left).
  • ☁️ Optional Gemini second opinion when online. The app never depends on it.

How we built it

The classifier. We started from a MobileNetV2 checkpoint fine-tuned on Planracy). We converted it PyTorch → ONNX → onnx2tf → TensorFlow Lite so it runs on the phone through react-native-fast-tflite. The app has to reproduce the training preprocessing exactly: center-crop, resize to $224 \times 224$, then normalize each pixel:

$$x' = \frac{x}{127.5} - 1 \quad \in [-1, 1]$$

The model outputs raw logits, so we apply softmax in the app to get a confidence score:

$$p_i = \frac{e^{z_i}}{\sum_{j=1}^{38} e^{z_j}}$$

The offline assistant. SmolLM2-360M-Instruct (a GGUF file of about 370 MB, an optional one-time download) runs on the phone's CPU through llama.rn. To stop a small model from making things up, we ground every answer in our own treatment data. We embed each of tthe same already-loaded model, cache the vectors in SQLite per language, andretrieve the closest matches by cosine similarity:

$$\text{sim}(q, d) = \frac{q \cdot d}{\lVert q \rVert \, \lVert d \rVert}$$

A keyword-overlap fallback covers the time before that index has finished building. Voice input uses Whisper tiny (whisper.rn). Answers are read aloud with the phone's built-in text-to-speech.

Everything else. Expo SDK 57 + React Native, Expo Router, SQLite for scan h for translations. Google ML Kit's bundled on-device text recognizer handleslabel OCR. The dosage maths scales a typical label rate $r$ by plot area $A$ (or plant count): $Q = r \cdot A$.

Challenges we ran into

  • Android can't record WAV. Whisper needs raw WAV audio, but Android's MediaRecorder can't produce it. We captured raw PCM with a separate audio module and wrote the WAV header by hand.
  • Fitting three AI models on one phone. A classifier, a chat LLM and a speech model all run locally. llama.rn only uses the GPU on iOS, so Android inference is CPU-only. We kept the LLM prompt to a few hundred tokens, capped history at 6 turns and made the big
  • Model conversion on a small laptop. The official ai-edge-torch route pulled in a huge dependency chain. We used ONNX → onnx2tf instead, and capped the Android build's CPU and RAM so an 8 GB machine could still compile whisper.cpp from source.
  • Small-model output. SmolLM2 answers in markdown and sometimes hits the token limit mid-sentence. We wrote a small markdown renderer, trimmed cut-off answers back to the last complete sentence, and removed markdown before text-to-speech.
  • Edge-to-edge Android quirks. On newer Android the keyboard covered text inputs, and Android cleared the camera cache before the Gemini request could read the photo. We fixed these with a custom keyboard-aware scroll view and by saving each photo to permanent
  • OCR false positives. A pesticide label prints the bottle's volume and the application rate. Our parser only accepts a rate with an explicit "per litre" denominator so it doesn't grab the wrong number.

What we learned

  • On-device AI is practical on a mid-range Android phone today, with no server,leaving the device.
  • A small model plus good retrieval beats a bigger model with no grounding for a narrow domain like crop treatment.
  • Matching the training preprocessing exactly matters more than almost anythingne.
  • "Offline-first" has to be designed in from the start. Every feature needs a working answer when there's no network.

What's next

  • Test with farmers and agricultural extension officers in the field.
  • Have native speakers review the 10 machine-translated languages.
  • Train on real field photos (the model was trained on single leaves against plain backgrounds).
  • Add more crops and regional product databases for dosage.

Try it out

Built With

Share this project:

Updates

Submission history