Pollensight

Capture bees, flowers, and paths across various videos.
RL-backed MuJoCo model for a robotic pollinator's path given a random plot.
Attributes collected across videos.
Forecast paths given the path a bee has taken so far.
Natural language search to jump to any moment of a video.
Results & Accuracy
Feature Overview
Architecture Diagram
Technology Stack
Heatmap of flowers that were pollinated.
Upload your own video to analyze and predict!
Dataset breakdown.
Perception information for a robot to pollinate a plot accurately.

Inspiration

Pollinators are disappearing, and about a third of the food we grow depends on them. We wanted to turn camera footage into something actionable: which flowers got visited, which didn't, and how a pollinator robot could achieve perception.

What it does

Pollensight turns ordinary fixed-camera video into a per-flower record of insect visits. It can aslo forecast where an insect will go next and plans a robot route to the flowers it missed.

Video: tracks insects with boxes and trails. Flower boxes light up during a visit, and a timeline shows every visit.
Search: plain-English natural-language search across clips ("when does the bee land on a flower?") that jumps the video straight to that moment.
Analytics: visits, dwell time, transit time, flight speed, revisits and a heatmap of where the insect spent its time.
Forecast: the insect's predicted path for the next 2 seconds, shown as a spread of possible futures, plus the odds for its next flower and how much longer it will stay.
Field: on farm clips, flowers are colored by how many visits they still need, with a robot route that covers the most of them within a time budget.
Dataset: accuracy across all 48 clips, with the visual audits behind every number.
Upload: analyze your own fixed-camera video.
Simulations: a bee trained with reinforcement learning flies over a random flower field it has never seen, in interactive 3D. The same flight is rendered in MuJoCo, with a Crazyflie 2 drone standing in for the bee.

How we built it

Pollensight is a four-stage pipeline.

Perception (Python, OpenCV, NumPy): OpenCV background subtraction and connected components find moving blobs, and a dynamic-programming search links them into flight paths. Every half second we send a 160 px crop to Perceptron's Mk1.5 vision model to check that there's really an insect there, and we keep a path only if at least 70% of its crops are confirmed. Mk1.5 also finds flowers in a median frame, which gives us stable flower IDs for detecting visits.
Perceptron API layer: we wrote our own client on top of the perceptron SDK. It uses box, point and clip annotations, JSON-schema output and the Files API for video. Every answer is cached under a SHA-256 key, so the whole project replays offline for free.
Intelligence:
- Forecasting (scikit-learn): a k-nearest-neighbors "path fan" for the next 2 seconds, gradient-boosted next-flower prediction and Monte Carlo rollouts. These are trained on 2,335 real insect flights and tested on camera stations the model never saw.
- Route planning: an orienteering planner (greedy insertion plus 2-opt) that picks the most valuable flowers to visit within a time budget.
- Simulation: a Gymnasium environment with a Stable-Baselines3 PPO bee, replayed in MuJoCo using the Crazyflie 2 model from MuJoCo Menagerie.
Web app (React 19, TypeScript, Vite 8): Canvas 2D and SVG overlays, a three.js 3D Simulations page, and custom Vite middleware that streams video with HTTP range requests and runs the Python forecast, analysis and simulation behind API routes.

Data: 48 clips from two Monash University datasets (CC BY 4.0): 38 honeybee close-ups and 10 strawberry-farm clips from overhead cameras.

Accomplishments that we're proud of

	Result
Insect tracking (48 clips)	90% of the path covered, median error ≈ 3–4 px
Visit detection (farm clips)	90% of detected visits are real (64% without Mk1.5)
Flower detection	98% precision, 87% recall
Next-flower prediction (unseen cameras)	right flower in the top 3 picks 79% of the time (nearest-flower guess: 68%)
Robot route	delivers 39–56% of missing visits in 30 s (random order: 17–33%)

We also built a full pipeline that anyone can reproduce: 45 tests run offline and 144/144 data files pass validation. The whole project goes from raw video to a 3D drone simulation in one app.

What we learned

This project was our way into physical AI. None of us had built AI that acts in the real world before, and we learned how to do it by using computer vision to turn raw video into data that a forecaster, a route planner and a robot can act on.
Pairing a cheap classical tracker with a strong vision model that checks its work beats either one alone. Mk1.5 raised visit precision from 64% to 90%.

Tech architecture

 Field video ──► PERCEIVE ──────────────► JSON contract files ──► UNDERSTAND ─────────────► EXPLORE
 (mp4, fixed     OpenCV motion tracking   tracks/  field/         Analytics (NumPy)          React + TypeScript
  camera)        DP path linking          plan/    search/        Forecast (scikit-learn)    Vite dev server
                 visit detection          (validated)             Route planner              /api/* routes
                        │                                         RL bee (Gymnasium + SB3)   three.js 3D page
                        ▼                                         MuJoCo replay
                 Perceptron Mk1.5
                 (cached client:
                  crops, flowers)

Layer	What it does	Tech
Capture	Clips and public datasets	Fixed-camera mp4 (720p, 30 fps), Monash datasets via figshare, GitHub Releases, ffmpeg / imageio-ffmpeg
Perceive	Insects, flowers, visits	Python, OpenCV (median background, connected components), NumPy (dynamic programming)
Vision model	Confirms insects, boxes flowers	Perceptron Mk1.5 via the `perceptron` SDK: box detection, JSON-schema output, Files API
Understand	Analytics, forecasts, plans, simulation	NumPy, scikit-learn (k-NN, gradient boosting), pandas, joblib, SciPy, Gymnasium, Stable-Baselines3, MuJoCo + Menagerie Crazyflie 2
Explore	The web app	React 19, TypeScript, Vite 8, Node.js, Canvas 2D, SVG, three.js
Quality	Keeps it correct and reproducible	pytest, JSON contracts + validator, accuracy evals, visual audits

Challenges we ran into

Tiny, fast insects on busy backgrounds. Plain motion tracking picked up leaves moving in the wind, and running a vision model on every frame was too slow and too expensive. Our answer was a hybrid: cheap OpenCV tracking proposes paths, and Mk1.5 checks samples of them.
Measuring accuracy honestly. We compared against each dataset's own reference tracker and visually audited the flower detections, rather than trusting results that just looked right.
Four people editing overlapping code at the same time. Agreeing on fixed JSON formats (CONTRACT.md) and a validator early on kept everyone's work compatible.

What's next for Pollensight

Real hardware: run the planned routes on an actual drone or ground robot in a test plot.
Live video: process camera streams in real time instead of uploaded clips.
More pollinators: support more insect species and crops beyond honeybees and strawberries.
Grower dashboard: pollination coverage across many cameras and over a whole season.
Better simulation: move from replaying recorded paths to flight with real physics, and train the RL bee on real farm layouts.