FlyBot on AWS

Trap cameras that look before they count: OpenCV 5 and a guarded agent on AWS for fruit-fly monitoring.

Comment

Try it: https://flybot.agrhub.com/auth/login, press Guest access: a shared, read-only demo account for two hours (the two real traps, a labelled simulated farm network, the Sentinel console and the chatbot).

Code: https://github.com/agrhub/flybot-aws (submodules https://github.com/agrhub/flybot and https://github.com/agrhub/camos-esp32-trap)

Team and advisor: AGRHUB, makers of FlyBot. Agronomy advisor: MSc Trương Quốc Ánh, Institute of Agricultural Science for Southern Vietnam (IAS, under VAAS); the advisor is not a Devpost team member.

Inspiration

Fruit flies are "by far the most important group of horticultural pests worldwide" (De Meyer et al., ZooKeys 2015); in Africa alone they cost about US$2 billion a year (icipe). We make FlyBot, which photographs fruit-fly traps in orchards and counts the flies with AI. Our data showed what a count misses: one camera filmed a building and every "fly" it reported was false; glare and blur were counted like any photo; the camera never reacted. Growers need a system that notices a bad photo, asks for a better one, and bothers them only when it matters.

What it does

Every trap photo, from EZVIZ cloud cameras or from our open ESP32-S3 trap camera, goes to an OpenCV 5 Lambda on AWS. It counts the flies with YOLO26n on OpenCV 5's new DNN engine, grades the image quality, finds the trap, registers the photo with the previous one to separate new flies from old ones, measures crowding, and detects hands, people and a moved camera (the Lambda stores no picture; FlyBot blurs the face regions it returns). A small agent then decides what should happen next under guardrails: take another photo, change the camera's image settings or schedule, ask a person to review, or alert the grower. FlyBot shows each decision as a confirm card in the app and on Telegram; nothing reaches a camera until a person presses Confirm, and the new photo starts the loop again. Growers can also ask FlyBot's chatbot about a photo ("why did the count jump?"): it calls the same Lambda in deep mode, where a Claude Haiku 5.5 planner answers from OpenCV's numbers, never from the pixels. ESP32-S3 cameras join on their own (zero-touch registration through AWS IoT Core), are claimed by QR code, stream live view over Kinesis Video Streams WebRTC, and are managed as a fleet (firmware channels, OTA rollouts with a battery guard, remote log, crash reports). A battery camera that sleeps between photos still takes orders: FlyBot keeps a live request, a GPS fix or a setting for its next wake and shows when that is; Wi-Fi or 4G is chosen remotely. Judges get Guest access, a labelled simulated farm counted by the real Lambda, and reports of their own.

How we built it

  • OpenCV 5: the new DNN engine (YOLO26n ONNX, NMS, box rules; the engine actually used is recorded), imgproc (HSV, contours, perspective, Laplacian, histograms, adaptive threshold, watershed), features (SIFT + FLANN + USAC MAGSAC; ALIKED + LightGlue on demand), MOG2, phaseCorrelate, YuNet face regions, and 2 x 2 tiles for small flies on sticky cards.
  • Agent: a deterministic rule planner for every photo and an LLM planner for questions, 14 typed tools, guardrails (run states, budgets, quiet hours, battery, evidence-grounded proposals, untrusted-text filter, kill switch) and a trace for every photo.
  • AWS (one SAM stack, Sydney): Lambda arm64 container built by CodeBuild into ECR, EC2 t4g.micro running FlyBot (Node, Vue, MongoDB, Caddy TLS), IoT Core (thing type, per-thing policy, shadow, RPC, presence), Kinesis Video Streams WebRTC, S3, Secrets Manager, CloudWatch alarms. One command deploys it: ./deploy.sh.
  • Device: ESP-IDF firmware for an ESP32-S3 board with an OV5640 wide-angle lens, 4G, GPS and a solar-charged LiPo cell in a hanging trap (about US$39 of parts, camos-esp32-trap docs/BOM.md); deep sleep, a setup portal, a web console, OTA updates and a remote log over IoT Core.

Challenges we ran into

YOLO26n's end-to-end head still left duplicate boxes and counted the trap's lens, so we added NMS and fruit-fly box rules. Live video from a microcontroller without a hardware H.264 encoder needed the camera's own JPEG frames over the WebRTC data channel. Our orchard traps were empty for most of October, so we generated labelled synthetic photos with known counts; they showed that flies on a sticky card shrink below 20 px when the photo is letterboxed to 640, and tiling the frame took recall from 0.07 to 0.82 without retraining. And COOL turned out slower than the plain pip wheel on our DNN-heavy workload.

Accomplishments that we're proud of

  • YOLO26n on the OpenCV 5 DNN engine: test mAP@50 0.953 and mAP@50-95 0.700, with detections identical to ONNX Runtime; the box rules raised F1 on the held-out test split from 0.8788 to 0.9006.
  • 38 of 38 failure-injection runs passed with zero false actuations (image defects, prompt injection, LLM and camera failures).
  • Our agronomy advisor judged 43 of the 49 agent decisions he could decide correct (87.8 %, 95 % interval 75.8-94.3 %, 1 unsure); the 6 he rejected are needless recaptures on one camera. We acted on it: lure traps now get their own glare limit, and on the same photos 1 of those 7 recaptures remains (the unsure one) while none of the 43 correct decisions changed.
  • One-command AWS deployment: about 17 USD a month for the demo, 0.046 USD per 1,000 photos, warm Lambda p50 980 ms.
  • An honest COOL benchmark on 500 real photo pairs.
  • Guest access with a labelled simulated network of 114 traps on 47 farms; its 2,852 seed photos went through the real Lambda.
  • A judge can run 24 real orchard photos through the Lambda's handler with one command, no hardware and no AWS.

What we learned

A vision system earns trust by refusing to count bad photos and showing its evidence. ENGINE_AUTO can silently fall back to the classic DNN engine, so check it. Synthetic photos with a known truth find weaknesses fast, but they are not field data. A person on every action keeps the agent simple and trusted.

What's next

Ask our agronomy advisor to review the quality gate's new glare limit on fresh photos (we re-scored his photos, he has not re-reviewed them); label 150 trap photos (ready, 0 done); improve recall on small flies in full lure-trap views; measure the ESP32-S3 trap's battery life in the orchard and fix its weaker Wi-Fi on battery; add yellow sticky cards; and try OpenCV 5's in-library VLM for second opinions on doubtful photos.

Built With

Share this project:

Updates

Submission history