HerdSense

stress detection that sees what satellites can't, running right on your phone.

Comment

Inspiration

Livestock stress signals appear days before they are seen from space. Animals bunch tighter, drift into erratic motion, and drop their heads. But the farmers who could act on those signals rarely have access to satellite feeds, wearable collars, or farm-scale monitoring hardware. We wanted to turn the one device every herder already owns, a phone, into a livestock stress scanner. No cloud, no collars, no cost. If the detection and analysis could run entirely on the device, the barrier to actionable animal welfare data would essentially disappear.

What it does

HerdSense is an offline-first, browser-based edge AI platform. Point your phone camera at the herd, record 20 seconds of video (or upload a clip), and HerdSense returns a clear 0-100 stress score within seconds. Under the hood, a YOLOv8-Nano model running in the browser detects livestock frame by frame, an IoU tracker assigns consistent identities, and a fusion engine blends three behavioral signals into one score with a plain-language recommendation. Those signals are clustering (Inter-Animal Spatial Index), motion (speed variance plus direction changes), and posture (aspect-ratio drift that captures head droop). The raw video never leaves the device. When a scan finishes, the report auto-shares over a WebRTC mesh to nearby farmers in the same geohash region, and the map clusters alerts when 3 or more high-stress herds appear within 15km. It is a full PWA.. installable, offline, with history persisted in the browser.

How I built it

I started from the user flow first: a React + TypeScript app with six screens (Home, Camera, Upload, Analysis, Map, History), then layered the ML pipeline on top. Detection is YOLOv8-Nano exported to ONNX and run through ONNX Runtime Web with WebGL acceleration and a WebAssembly fallback. Frames are sampled at 2fps, drawn to a 640x640 canvas, converted to a planar RGB float tensor, and passed to the model. Non-Maximum Suppression cleans up duplicate boxes. A roughly 90-line greedy IoU tracker (instead of ByteTrack or DeepSORT, since livestock look alike, so appearance models add little for a 20-second window) assigns IDs, and centroid displacement (instead of expensive optical flow) measures motion. For the network, we used PeerJS to wrap WebRTC signaling, with no server to run and data flowing peer-to-peer. Leaflet + OpenStreetMap render the stress map (free, no API key, works offline), and geohash rooms group nearby devices. Throughout, we made deliberate trade-offs: localStorage over IndexedDB (small scores, not video), Haversine over H3/DBSCAN, Vite over Next.js.

Challenges I ran into

The hardest problems were all at the browser-ML boundary. Tensor parsing: the YOLOv8 ONNX output layout (1, C, N vs 1, N, C) had to be handled generically or nothing would parse. RGB channel packing: an off-by-layout bug in the planar preprocess silently corrupted every frame. Video seeking: seeking through a video element to sample frames deadlocked the pipeline, so we built an explicit waitForSeek with a timed fallback. Class ID filtering: the model's COCO output had to be mapped to livestock classes only (cattle, sheep, horse), since a mis-filter meant sheep and horses never registered. WebGL's silent failures: on real devices, WebGL EP can return NaN output without throwing an error, which produced "no animals detected" with zero feedback. We added first-frame output validation and an automatic fallback to the reliable wasm EP. Production deployment: the 26MB wasm runtime and 13MB model needed careful asset handling. The service worker pre-cached them for offline, and we had to swap a slow /wasm/ path for hashed build assets. Plus the usual integration grief: iOS Safari MediaRecorder mime types, PeerJS connection drift, localStorage quota crashes on mobile, and DOM memory leaks from detached video elements.

Accomplishments that I'm proud of

  • A genuine zero-infrastructure system: no backend, no database, no accounts, yet reports still reach nearby peers and drive a live regional alert map.
  • True on-device privacy: the video never leaves the phone; stress data travels only device-to-device.
  • The scoring engine itself: fusing three biomechanical signals into a single validated 0-100 score with human-readable recommendations.
  • Surviving the browser-ML gauntlet, from ONNX tensor layouts to WebGL's silent NaN, and shipping a build with 23 passing unit tests, strict TypeScript, WCAG keyboard accessibility, and a documented edge-pipeline guide.

What I learned

That the browser is a legitimate (and underrated) ML inference target, GPU-accelerated via WebGL and WebAssembly, and that "it runs in Node" tells you almost nothing about whether it runs on a phone. We learned that ONNX Runtime Web's execution providers have sharp edges: WebGL can silently return garbage where wasm is deterministic, so correctness needs runtime validation, not just try/catch at load time. We learned to respect the platform: video seeking is async and unreliable, so you design for timeouts; mobile storage is tiny, so you plan for quota errors; iOS Safari breaks rules, so you test the actual device. And we learned the value of ruthless simplification. Choosing a greedy IoU tracker over DeepSORT and centroid displacement over optical flow got us 90% of the signal at a fraction of the complexity.

What's next for HerdSense

  • Better models: fine-tune YOLOv8 on livestock datasets (goats, donkeys, camels) to go beyond the stock COCO classes and improve per-breed accuracy.
  • Multimodal stress: actually sample ambient audio distress (the fusion engine already reserves a weight for it) to catch vocal stress alongside visual cues.
  • WebGPU: move inference to the WebGPU execution provider for a big speed and battery win on modern phones.
  • Smarter networking: full peer-relay for regions with sparse peers, and persistent scan history shared across a herd rather than just the current session.
  • Field validation: partner with smallholder farms to calibrate the scoring scale against veterinary ground truth and fine-tune recommendations per species and season.

Built With

Share this project:

Updates