Frequency Vision

A bird and a drone look the same at 30 fps. Only the drone hums. Event cameras + OpenCV 5 on COOL/Graviton4 confirm drones by propeller hum: 25% fewer false alarms, same recall.

Comment

Inspiration

A drone is a dot. So is a bird. At airports, prisons and stadiums, the hard part is not seeing the dot. It is knowing what it is. In December 2018, reported drone sightings shut Gatwick's runway for about 33 hours, cancelling about 1,000 flights. The FAA receives more than 100 reports of drones near US airports every month (FAA).

Every drone carries a signature no bird has: spinning blades. They pass the camera hundreds of times a second:

$$f_{bp} = N_{blades} \cdot f_{rotor} \approx 200\text{–}700\ \text{Hz}$$

A 30 fps camera can only represent 15 Hz, so a 316 Hz rotor folds down to a meaningless 14 Hz. An event camera fires per pixel with microsecond timing, so the propeller writes its exact pitch straight into the data. Frequency Vision listens to it.

What it does

Look → Listen → Decide, live, on real drone recordings:

  1. Look. A YOLOv8n detector running in the OpenCV 5 DNN engine (ENGINE_NEW) proposes drone boxes in the 30 fps RGB frame.
  2. Listen. Inside each box, the last 100 ms of events become a 10 kHz signal. OpenCV 5 cv.dft finds the blade-pass peak, which is scored against its local noise floor. A drone hums, and a bird, a cloud or a rooftop stays flat.
  3. Decide. An agent on Amazon Bedrock (Claude Sonnet 4.6) reads the vision results and chooses the next measurement: a longer window, a tighter box, a whole-frame frequency map, a Kalman track, or asking a human before any critical alert.

All of it runs on COOL (Cloud Optimized OpenCV) on AWS Graviton4, live at https://100-27-6-111.sslip.io.

Results on 84 minutes of real drone video

All numbers come from FRED, a public event + RGB drone dataset. Every threshold was picked on train and reported on all 47 test sequences: 151,768 frames, 84.3 minutes.

Detector only + Frequency Vision hum check
False alarms per minute 412.7 308.4 (−25%)
Precision 0.670 0.731
Recall 0.544 0.544 (unchanged)
True detections lost – 26 of 70,794
  • 8,790 false detections removed while keeping recall. A stricter setting reaches precision 0.815.
  • The physics checks out. Our measured hum per drone model matches independent PropPulse estimates: DJI Mini 2 320 vs 316 Hz, Tello 564 vs 539 Hz, Mini 3 245 vs 223 Hz.
  • The hum separates drones from background: AUC 0.917 against random patches.

The agent: vision drives every decision

  • 24 FRED scenarios, 16 of them held out from every fix:
    • Bedrock agent: 12 of 24 solved, 0 false alerts.
    • Hand-written rule planner: 10 of 24, with 1 false alert.
    • Fixed schedule: 9 of 24.
  • Blind the vision and the agent fails. With the OpenCV results hidden or shuffled, Bedrock drops from 12 to 6 of 24. In 30% of steps the agent's next action changes when vision is hidden, and in 41% when it is shuffled. The agent is reading the camera, not guessing.
  • It finds what the detector misses. In FRED sequence 142 the detector saw nothing. The agent mapped the whole frame, found one spot humming at 545 Hz, confirmed it, tracked it, asked a human and alerted. It was a DJI Tello.
  • Humans stay in charge, and the code enforces it:
    • operator rejects → 0 alerts in 6 of 6 runs;
    • operator never answers → the run ends "unconfirmed", 0 alerts;
    • a deterministic guard blocks any alert without hum evidence on that box.
  • 21 of 21 injected faults handled, 0 unsafe alerts. The faults covered Bedrock down, timeouts, malformed tool calls, crashing tools, empty and corrupted event files, budget overruns, a rejected or silent operator, and bad MCP calls.
  • Open and portable. The same 8 guarded tools are exposed to Bedrock and as an MCP server, tested end to end. Every step is logged as JSON to S3.

COOL on Graviton4: verified live

  • Verified live. The live endpoint imports OpenCV 5.1.0-dev from /opt/cool with the KleidiCV 0.7.0 HAL. Press "Verify COOL" on slide 6 and the server proves it in real time: its own instance (c8g.medium), AMI, Marketplace product code, CPU features (SVE2, BF16), and live timings on a real drone event window.
  • COOL speeds up our event engine. On real FRED event windows, on the same Graviton4 host, COOL runs the full-sensor frequency map up to 1.45x faster than stock pip OpenCV 5 (149.9 vs 217.9 ms per 500 ms window, 4 threads), with identical outputs. The gain grows with the window length (1.26x at 100 ms), exactly where event cameras need it. In the single-operation tests: GaussianBlur 1.89x, dft 1.54x.
  • 30% cheaper than x86. End to end (detector + hum check + frequency map), Graviton4 + COOL costs 30 to 33% less per window and runs 17 to 22% faster than an x86 c7i.xlarge. On 500 ms event windows it is 2.9x faster than x86. Prices are from the AWS Price List API.
  • We tuned for COOL. Switching the detector to OpenCV's classic DNN engine cut its time under COOL from 103.5 to 55.0 ms, with every box and score unchanged.
  • Reproducible. One command deploys the COOL endpoint, and one runs the same-host benchmark. The full write-up for judges is in docs/COOL.md.

How we built it

  • OpenCV 5 everywhere:
    • calcHist, createHanningWindow, getOptimalDFTSize, dft (including one DFT_ROWS call for up to 20,000 per-pixel spectra), magnitude, minMaxLoc;
    • GaussianBlur, Otsu threshold and connectedComponentsWithStats for the rotor mask;
    • KalmanFilter and moments for tracking;
    • DNN readNetFromONNX, blobFromImageWithParams and NMSBoxes. The ONNX model matches PyTorch to 4.6e-4.
  • AWS:
    • EC2 Graviton4 from the COOL Marketplace AMI, behind Caddy HTTPS;
    • systemd plus a health timer;
    • SSM instead of SSH, and a least-privilege IAM role;
    • CloudWatch logs and an auto-recover alarm, S3 traces, Bedrock and AWS Budgets;
    • all deployed with one script.
  • Live page: six swipe slides. Every chart is computed live on the COOL server: the spectrum, the detector, the maths, the agent and the COOL check.

Challenges we ran into

  • We found a silent OpenCV 5.0.0 bug. The Python binding cuts NumPy int64 arrays to their low 32 bits, so microsecond timestamps break with no error. We built a minimal repro, worked around it, and wrote an upstream issue.
  • Lights hum too. Mains-powered screens flicker at multiples of 100 Hz. A 100 Hz notch and a whole-frame frequency map (one frequency everywhere means lighting, not a rotor) cut the busiest-patch false alarms from 4.3% to 2.6%.
  • Getting COOL's own Python onto the live path, then proving it on every request.

Accomplishments that we're proud of

  • A new signal for drone confirmation: the propeller's own pitch, measured with OpenCV 5 on real event-camera data.
  • −25% false alarms at the same recall, on every frame of the 84-minute test split.
  • An agent that measurably uses vision. Its success halves when vision is blinded. It has 0 false alerts and a human gate enforced in code.
  • COOL verifiable by anyone, live, in one click.
  • An OpenCV 5 bug found and documented for upstream.

What we learned

Seeing time, not just space, turns a hard recognition problem into a measurement. Agents earn their place when every tool is a real measurement, and a guard and a human sit between the agent and any alert.

What's next for Frequency Vision

Every machine that spins has a voice. We are building the camera that hears it.

  • From "a drone" to "which drone". Each model already hums at its own pitch (Mini 2 at 320 Hz, Mini 3 at 245 Hz, Tello at 564 Hz). Next is a frequency fingerprint library that names the make and model from the hum alone, and reads rotor speed changes as a sign of a heavy payload or a hard manoeuvre.
  • Spiking neural networks on neuromorphic chips. Event cameras speak in spikes. A spiking network that reads the hum natively can run at milliwatt power on the sensor itself, so every camera mast becomes a drone sensor with no server behind it.
  • COOL at the edge. The same OpenCV 5 + COOL stack runs on any Arm device, so one Graviton4 instance in the cloud and Arm boxes on the mast run identical code.
  • Many cameras, one sky. Two or more event cameras hearing the same pitch from different angles can locate the drone in 3D and follow it across a whole site.
  • Beyond drones. Fans, motors, pumps, wind turbines and bridges all vibrate at telltale frequencies. The same engine becomes a contact-free health monitor that spots a failing bearing before it breaks.
  • Give it back to OpenCV. We will contribute the hum engine as an OpenCV module and send the int64 binding fix upstream, so every OpenCV user can measure frequency from events.

Built With

Share this project:

Updates

Submission history