DoorSight: Accessible Ring Doorstep Care

A Ring-powered doorstep assistant for elderly and low-vision residents: it watches doorbell video, tracks packages and odd-hour visits, and sends calm, accessible updates to residents and caregivers.

Comment

Inspiration

A video doorbell assumes you can see the video. For an older or low-vision person living alone, the questions are simpler and harder: did a parcel arrive, is it still outside, who is in the driveway at 3 AM? And the family member who looks out for them doesn't want every motion alert; they want to know when something is actually wrong.

What it does

Every Ring event (package, vehicle, motion) goes to an agent. It opens the doorbell's live video, works out what is at the door, and remembers what happened before.

  • For the resident: a large-text, high-contrast, screen-reader-first view. "A package was left at your door at 2:10 PM", a gentle reminder three hours later, and one big "I picked up the package" button.
  • For the caregiver: short, evidence-based email alerts:
    • Possible missing package: only when the same camera view shows the package gone and nobody marked it picked up.
    • Unusual-hour activity: a car at 3 AM, scored against the home's own visit pattern.
    • Daily digest: one summary instead of a stream of notifications.
  • Every alert shows why: the score, the camera-view check, and each tool call the agent made. Every message is labelled with where its information came from (vision model, automatic estimate, or rule).

How I built it

  • Ring: GET /v1/devices for discovery; live video over WHEP captured with aiortc (video-only, 1 frame/s, sessions always closed cleanly); HMAC-verified webhooks; one-way account linking (nonce matching, App-Integrations POST + PATCH, rotating refresh tokens).
  • Vision: YOLO-World open-vocabulary detection runs locally and finds the "cardboard box" a standard COCO model misses. Claude Haiku 4.5 on Amazon Bedrock describes the scene as validated strict JSON.
  • Agent: a Strands agent with eight tools (capture, describe, package state/update, visit baseline, notify resident/caregiver, daily digest).
    • Safety rules live in the tools, not just the prompt: a capture from a different camera can never mark a package missing. Each camera view is fingerprinted with a perceptual hash and histogram, with thresholds measured on real captures.
    • If the model fails, a deterministic rules brain finishes the event through the same tools.
  • AWS: DynamoDB for state, S3 for private snapshots (presigned URLs), SNS for caregiver email, and a least-privilege IAM user so the server never runs with root credentials.
  • Web: React + Vite, 0 axe-core accessibility violations (WCAG 2.2 A/AA).
  • Tests: 262 pytest tests: both stores (SQLite and DynamoDB via moto), the agent loop driven by a scripted fake model, and account linking against a fake Ring OAuth/API server.

Challenges I ran into

  • No live webhooks: the Ring Playground's buttons switch the video clip but send no webhooks, so I added /simulate-event, which runs the same payload through the same handler.
  • Linking needs a paid plan: live account linking stops at the Connect step without a Ring Protect plan, so I verified the full flow against a fake Ring server.
  • Bedrock blocked: my AWS account can't invoke Bedrock models yet (support case open). The agent runs on its rules brain and switches to Haiku automatically once access works.
  • Short sandbox streams: WHEP streams sometimes end after 10–15 s with no signal, which needed careful capture logic.
  • A false alarm by design: a capture from a different camera would report a delivered package as missing. That led to camera-view fingerprinting.

Accomplishments that I'm proud of

  • Safety enforced in code: refusals from the tools reach the model as errors.
  • Honest labelling: every alert explains itself, and an estimate is never passed off as a vision-model answer.
  • A resident view built for low-vision users, with 0 axe violations.
  • A scoped IAM user, with denials verified.
  • A 12-entry friction log with reproduction steps for the Ring team.

What I learned

  • How Ring's one-way account linking works (account-bound nonces, the mandatory PATCH, why one-way apps can't DELETE).
  • Why DynamoDB GSIs are the wrong tool for read-your-writes state.
  • How much it helps to design an agent so it can be tested without its model.

What's next for DoorSight

  • Switch the agent to Bedrock (Haiku 4.5) as soon as account access is granted.
  • Verify account linking against a real Ring Protect account.
  • Add contact sensors once the sandbox offers them.
  • A scheduled daily digest and caregiver-set quiet hours.

Built With

Share this project:

Updates

Submission history