Worldline

See a whole camera-trap video at once: OpenCV 5 traces every animal visit through a 3D block of the clip, so you know what came, when and for how long without watching the footage.

Comment

Inspiration

Camera traps record far more footage than anyone can watch. Most clips are empty, or hold an animal for a few seconds, yet someone still has to scrub through each one to answer simple questions: did anything come by, when, for how long, and what was it? I wanted a way to take in a whole clip at a glance.

What it does

Worldline turns a video into a solid block: the picture's width and height, with time as the third dimension. Every animal visit becomes a coloured tube through that block. A tube's length is how long the animal stayed, and its path is where it went.

  • Explore the block. Drag the "now" sheet through time. Click a tube to jump to that animal. Switch between an angled view, a side view (each visit as a band along time) and a frame view that plays like a normal video with the traced outline on it.
  • Measure each visit. Entry and exit times, dwell time and distance moved, split correctly when a clip is several recordings joined together.
  • Search by word. Typing "deer" or "badger" lights up matching animals in every clip. The search is honest: a match counts as confident only if the word fits the animal better than 26 other animal names, so "red fox" reports no confident match where there isn't one.
  • Try your own footage. Upload a clip of up to 60 seconds. It's processed privately, reachable only through its own link, and the video is deleted once processed. People are never tracked; anything that looks like a person is pixelated.

How we built it

OpenCV 5 does the computer vision.

  • VideoCapture decodes and samples the clip.
  • MegaDetector v5a, a camera-trap animal detector converted to ONNX, runs through cv2.dnn. The classic engine was twice as fast on Graviton, with identical detections.
  • A median background with a per-pixel flicker threshold (absdiff, morphology, connectedComponentsWithStats) catches motion the detector misses, without being fooled by rippling water or leaves.
  • goodFeaturesToTrack and calcOpticalFlowPyrLK follow each animal between detections. Frame-difference statistics find the joins between recordings.
  • grabCut, findContours and moments trace each animal's outline in every frame.

Search uses EmbeddingGemma 2. Each tube's vector is the average of several OpenCV-cropped images of the animal, so species names come from what the crops show, never from the detector's labels.

The viewer is Three.js. The block is ray-marched in a shader: its haze shows each pixel's change from its average, so what lights up is what moved, not what is merely bright.

AWS:

  • One EC2 Graviton4 instance (m8g.large) runs the app in Docker, defined in CloudFormation.
  • CloudFront is the only way in; the security group admits only CloudFront's address range.
  • A private, encrypted S3 bucket holds the app bundle.
  • Logs go to CloudWatch, administration is through SSM (no SSH), and an AWS Budget alert caps spending.
  • Moving from Graviton2 to Graviton4, with bfloat16 for the embedding model, made processing about four times faster.

Challenges we ran into

  • Water, leaves and snow. They move constantly. A single global motion threshold either missed animals or tracked ripples, so each pixel got its own threshold based on how much it normally flickers.
  • Clips joined from several recordings. These broke tracks or merged different animals, so I added cut detection that splits visits at each join.
  • A general-purpose detector missed small, dim animals at night. Switching to MegaDetector, which was trained on camera-trap images, found them; for example, a deer that YOLOX missed completely.
  • The detector mistakes some animals for people. A badger in snow was called a "person" at 79% confidence. Pixelating people is therefore applied only to uploads, and the upload panel says so.
  • Keeping it honest. I froze the pipeline before scoring a held-out set, and reported the failures alongside the successes.

Accomplishments that we're proud of

  • Development set (14 clips): 20 of 21 animal visits found, and 85% of tubes are real animals.
  • Held-out set (6 clips, frozen before the pipeline saw them): 16 of 17 visits found, and the right species name comes first for 78% of tubes, in line with the development set.
  • Search: typing a species brings up a tube of that species first in 8 of 9 searches.
  • Evidence: the evaluation, labels, failure cases and the scripts to reproduce them are all in the repo.

What we learned

  • Classical OpenCV earns its place next to deep learning. Background modelling, optical flow and GrabCut filled the gaps between detections and turned boxes into outlines.
  • Measure on data you didn't tune on. The held-out set kept the numbers honest.
  • A search that says "I'm not sure" is more useful than one that always answers.

What's next for Worldline

  • Have ecologists verify the labels.
  • Batch processing of whole camera-trap deployments, with a summary across days.
  • Better handling of animals at the frame's edge, which the search model struggles with (red foxes).
  • Ignoring burnt-in captions and timestamps.
  • Exporting visits as CSV for ecology tools.

Built With

Share this project:

Updates

Submission history