LookTwice

Smart doorbell alerts from any camera: OpenCV 5 finds what happened, a 2B decider decides, and a vision model on Bedrock looks twice only when it is unsure. Live on AWS Graviton.

Comment

What it does

Doorbell cameras fail two ways: they alert for everything (cats, cars, shadows) and they go quiet when the picture is poor (a dark porch, a smudged lens, a parcel taken while no "person" was detected). LookTwice turns a doorbell or porch clip into a short list of decisions: alert, ignore, log, or look again.

  1. OpenCV 5 perceives. MOG2 motion with hysteresis, YOLOX-S through cv2.dnn, blur / dark / infrared-night checks measured on background pixels, cut and title-card detection, tracking of person-sized movement the detector could not name, and a before/after doorstep comparison for packages (COCO has no parcel class).
  2. Strands Decider 2B decides. Every event becomes one typed question with four answers and a probability for each.
  3. The decision changes what happens next. recheck sends the frames back to OpenCV (gamma LUT, CLAHE, bilateral denoise, zoomed crops, detector again) and asks the decider a second time. Confidence under 0.4 sends one enhanced frame to Amazon Bedrock Nova Lite. Only an alert wakes the language model (Nova Micro) that writes the message; a PolicyGuard intervention denies the notify tool on anything else.

Safety rule: the second look may raise an alert or calm a non-person event, but it can never silence an alert about a person. One frame often catches the moment after someone has left, so the decider's alert stands and Bedrock's answer is kept beside it for the owner.

The page plays the clip, lists every event with its decision, source, confidence and time, and lets you re-run the clip, look harder, ask the decider again, ask the vision model, or drop in your own clip.

A real trace (from a published clip)

Dog in a dark doorway: OpenCV says "too dark" (brightness 80.7) → Decider 2B: recheck 57% → OpenCV enhances 8 frames (brightness 103, blur score 1815 → 4455) and finds an animal in 8/8 → Decider 2B: ignore, but only 33% sure → Bedrock Nova Lite: ignore, "A dog is standing at the door" (1.96 s, 960 tokens) → no notification, every step kept in the event record.

Results

Ten public clips (8 Ring clips under CC BY, 2 Pexels dog clips), 35 events, OpenCV run once so every method decided the same events:

Method Visit clips alerted Pet clips kept quiet Phone alerts Bedrock calls
Hand-written rules 7/7 0/3 20 0
Decider 2B 7/7 2/3 15 0
Decider 2B + Bedrock when unsure (LookTwice) 7/7 3/3 14 9
Bedrock on every event 4/7 3/3 4 35

Sending everything to the big model was quiet but missed 3 of 7 real visits. LookTwice kept every visit, silenced every pet-only clip, sent 30% fewer alerts than the rules and asked Bedrock about 1 event in 4.

How it runs on AWS

One Terraform apply builds it: an EC2 t4g.xlarge (Graviton2, Arm) runs the OpenCV 5 pipeline, Decider 2B on CPU and the page behind nginx (method allowlist, body caps, per-visitor rate limits, uploads capped and removed hourly); Amazon Bedrock Nova Lite and Nova Micro; the published clips in a private, encrypted S3 bucket; logs and metrics in CloudWatch; admin through Systems Manager with no SSH port; an IAM role limited to the two Nova models and its own bucket.

I tuned the OpenCV pass for the Arm chip without changing a single result: frames are decoded on a second thread, YOLOX runs on its own thread, counts stay in OpenCV instead of numpy boolean indexing, and YOLOX loads on OpenCV 5's classic dnn engine (1.97× faster than the new engine on Graviton2). Measured on the server at first boot: the OpenCV pass over the 10 clips went from 177.1 s to 113.0 s (1.57×) with the same events.

Challenges

  • Edited YouTube footage has cuts and title cards a real camera never produces; the pipeline detects and skips them (false "blurry" events from title cards: 6 → 0).
  • Nova Lite silenced real person alerts from single frames in early runs; the may_override rule fixed it (visits alerted went back to 7/7).
  • Decider confidence is often modest (median 52%), which is exactly why the escalation line exists.

What I learned

A small decider plus a rule about when to ask the big model beats both "rules only" and "big model on everything", on accuracy, cost and speed. And the vision result has to change what the system does next, not just be described.

What's next

A detector with a parcel class (package detection is the weakest part: 11/18 on the labelled set), a scene-type check so non-doorstep clips are described neutrally, and a GPU option for live feeds.

Honest limits

Package detection misses small parcels and night scenes. A single-frame second look can miss a person who just left (hence the safety rule). Decider 2B takes 7–8 s per event on the Graviton CPU: fine for a drop box, not a live feed. Narration in the video is Amazon Polly. Independent project, not affiliated with or endorsed by Ring LLC.

Built With

Share this project:

Updates

Submission history