Traction

Traction is a tractor-mounted crop disease detection system

Comment

What Inspired Us Farmers lose an estimated $220 billion annually to crop disease worldwide. Most of that loss is preventable — if you catch the disease early enough. But traditional detection means pulling samples, sending them to a lab, and waiting days for results. By then, the blight has already spread three rows over. We kept coming back to one question: what if the tractor itself could see the disease? Farmers already drive every row of their fields. The tractor is right there. All it needed was eyes and a brain fast enough to keep up. When Qualcomm introduced us to the Snapdragon X Elite's Hexagon NPU, we realized we had our brain. A neural network running at the edge — on the device, on the tractor, in the field — with no cloud dependency and no cellular requirement. TRACTION was born from that realization. How We Built It We split into three tracks that had to eventually become one app. Sid trained a ConvNeXt image classification model on a corn disease dataset covering four classes: Healthy, Northern Leaf Blight, Cercospora Gray Leaf Spot, and Common Rust. We started with MobileNetV2 but switched to ConvNeXt mid-hackathon when accuracy on our validation set was meaningfully better. The model was exported to ONNX and runs via QNNExecutionProvider directly on the Hexagon NPU. Benji built two things: a real GPS disease detection dataset with 26,000+ field-collected tagged events, and a local LLM advisory backend using Ollama. When the camera detects a disease, Benji's backend passes the label and confidence to a locally running Llama model which returns a specific treatment recommendation — spray schedule, fungicide type, severity assessment — in seconds, with zero internet required. Christian built the full Streamlit dashboard: 6 tabs, WebRTC live camera feed with async inference, a Ghost Tractor simulator, Google Maps heatmap overlays, farm profile management, live session recording, and a hardware telemetry dashboard showing real CPU/GPU/NPU utilization pulled directly from the system. The hardest part was making all three work together. Three branches diverging on the same files, a missing method the UI was calling, and a data schema mismatch between Benji's GPS data and the app's internal format. Model inference runs on the NPU. The LLM runs on-device. GPS logging is local. The only cloud dependency is Google Maps satellite tiles — everything else runs fully offline. What We Learned None of us had worked with ONNX Runtime's execution providers before. Learning that QNNExecutionProvider is the bridge between Python and the Hexagon NPU — and that PyTorch simply does not route to the NPU on Windows ARM — was the most important technical insight of the whole project. We also learned what temporal smoothing buys you in a real deployment. Raw per-frame predictions from a dashcam are chaotic — motion blur, sunlight glare, leaf shadows. A majority-vote window over 7 frames transformed an unusable flickering readout into a stable locked-in detection. Simple and effective. Challenges We Faced Model instability on live video. Raw inference flickered between classes every few frames. The 7-frame majority-vote smoothing solved this completely. Three branches, one app.py. Christian's UI, Sid's ML backend, and Benji's GPS and LLM work all touched the same files. Merging without losing anyone's features was the most time-consuming part of the project. MobileNetV2 to ConvNeXt mid-hackathon. Switching architectures with hours on the clock is stressful. The accuracy improvement on our validation set was significant enough that we made the call. It paid off. Async LLM without freezing the UI. Ollama takes 2 to 4 seconds to generate a treatment advisory. Running it synchronously froze the entire app. A background worker thread solved this so the advisory populates without blocking the camera feed. WebRTC on a local network. Removing the dependency on Google's external STUN server while keeping the camera working required switching to local-only ICE configuration. Works fine for on-device use.

Built With

Share this project:

Updates