PlantMD

Inspiration

Plant stress detection in research settings is still largely a manual, time-intensive process. Our teammate Ajay Das is a Plant Science Master's student who runs a greenhouse and conducts research on tomatoes, hops, and citrus plants. His day-to-day involves visually inspecting plants for signs of stress — nutrient deficiencies, light damage, disease onset — and making judgment calls that directly affect experimental outcomes. At scale, this becomes a bottleneck: a single researcher can only visually assess so many plants per day, and subtle early-stage stress signals are easy to miss.

We saw an opportunity to build something that could augment that workflow. Rather than replacing expert judgment, we wanted to create a tool that gives researchers a fast, quantitative first pass on plant health — flagging stressed specimens for closer inspection and providing AI-assisted context on potential causes. The same tool could also make plant diagnosis accessible to everyday plant owners who don't have Ajay's expertise, making it useful across both research and consumer contexts.

What It Does

PlantMD lets you upload a photo of a plant and receive an instant health assessment with specific care recommendations. Under the hood, it works in two stages:

A custom-trained Vision Transformer (ViT) analyzes the image to classify the plant's stress level
The stress classification is passed alongside the image to Google Gemini, which generates a natural-language diagnosis explaining what's likely wrong and what you can do about it

The result is a combination of quantitative ML analysis and LLM-powered interpretation — giving users both accuracy and actionability.

How We Built It

Custom Vision Transformer

We fine-tuned a Vision Transformer on the AgricolaModerna Healthy-Stress Dataset (deep-plants/AGM_HS on Hugging Face), which contains labeled images of healthy and stressed plants. The ViT model learns to detect visual indicators of plant stress — discoloration, wilting patterns, leaf texture changes — and outputs a stress classification score.

The ViT architecture splits each input image into fixed-size patches, linearly embeds them, and processes the sequence through a transformer encoder. For an image split into $N$ patches, each patch $x_p^i$ is projected into a $D$-dimensional embedding space:

$$z_0 = [x_{\text{class}}; \; x_p^1 E; \; x_p^2 E; \; \cdots \; x_p^N E] + E_{\text{pos}}$$

where $E \in \mathbb{R}^{(P^2 \cdot C) \times D}$ is the patch embedding projection and $E_{\text{pos}}$ are learnable position embeddings. The classification token $x_{\text{class}}$ aggregates information across all patches through self-attention layers and is used for the final stress classification.

Gemini Integration

The ViT model's output alone tells you how stressed a plant is, but not why or what to do about it. We feed the stress score and the original image into Gemini's multimodal API, prompting it to generate a diagnosis grounded in the model's quantitative assessment. This ensures Gemini's recommendations are anchored to real model output rather than guessing from the image alone.

Pipeline Summary

User uploads photo
       ↓
ViT Model → Stress classification score
       ↓
Score + Image → Gemini API
       ↓
Natural-language diagnosis + care recommendations
       ↓
Results displayed to user

Challenges We Faced

Dataset limitations: The AGM_HS dataset provided a strong foundation but covers a limited range of species and stress types. Generalizing to the huge variety of houseplants people actually own required careful model tuning and relying on Gemini to fill knowledge gaps for species outside the training distribution.

Grounding LLM output: A key design challenge was making sure Gemini's diagnosis was actually informed by the ViT model's assessment rather than just riffing on the image independently. We designed the prompt pipeline to explicitly anchor Gemini's reasoning to the quantitative stress score, treating it as a primary input rather than supplementary context.

Holiday hackathon time constraints: We built this over a short sprint during the holidays, which meant making pragmatic tradeoffs between model training iterations, UI polish, and pipeline reliability.

Accomplishments

Built a fully functional end-to-end pipeline from photo upload to actionable diagnosis, deployed live at plantmd.net
Successfully fine-tuned a ViT on a specialized agricultural dataset for plant stress detection
Validated the tool on real plants — we discovered a money tree had sun damage from being too close to a window, which PlantMD correctly identified
Designed an architecture that combines custom ML with LLM reasoning, where each component adds value the other can't provide alone

What We Learned

How to effectively chain custom model outputs into LLM prompts to get grounded, interpretable results rather than pure hallucination
The critical role dataset quality and scope play in specialized vision tasks — and how to design around dataset limitations
How Gemini's multimodal capabilities can serve as an interpretability layer on top of traditional ML classifiers

What's Next

We plan to expand the training dataset significantly so PlantMD can be used by university research groups and plant science labs as a tool for large-scale stress detection and hyperspectral image analysis. With a broader dataset covering more species, stress types, and imaging modalities, the same ViT pipeline could support agricultural research at scale — moving from a consumer tool to a scientific instrument.