Doorstep

A Ring doorbell that says what it sees — and refuses, out loud, when the frame cannot be described.

Comment

Inspiration

A Ring notification says "motion detected." For a blind or low-vision person that is not information — it is a prompt to guess. Someone leaving a parcel, someone standing at the door, and someone walking past the driveway all produce the same alert, and the only way to resolve it is to look at a picture.

Ring already knows something happened and already has the frame. What is missing is the sentence.

What it does

Doorstep turns a Ring event into an immediate, objective spoken description — and refuses out loud when the frame cannot be described.

Measured against the live Ring API, not asserted. An authenticated probe of the Ring Developers Playground on 2026-09-14 returned HTTP 200 on six endpoints — /devices, /locations, /users/me, and a device's /capabilities, /status and /configurations — each answering with server: envoy and a distinct x-request-id. The sandbox device is a Doorbell Pro reporting online: true. Full request and response detail: docs/00-research/ring-live-api-evidence.md.

Why the demo video shows a red "No Valid Ring Playground Token" banner throughout. Playground sandbox tokens expire after 30 minutes, and a live token must never be committed to a repository or recorded into a video. With no token present the app blocks every live call with a real HTTP 401 and fabricates nothing in its place — that refusal is the feature, so it is the honest state to record in. The six 200s above are the same API, reproduced with a fresh token and written up with headers.

  1. Webhook normalisation. The Ring Partner API sends no bounding boxes and no object labels, only a coarse sub_type inside data.attributes. Doorstep normalises that schema: it extracts human / vehicle / other_motion from motion alerts and maps legacy ding signals to button_press.
  2. Watermark excision. Under Ring's current spec every frame carries a mandatory watermark — logo top-left, device timestamp top-right. A multimodal model reads that text aloud instead of describing the porch. Doorstep's pure-JavaScript cropper excises the top 15% of rows — 134px on an 896px frame — and a unit test asserts that none of those rows reach the inference payload.
  3. Description. The cropped frame goes to Amazon Bedrock Nova Pro (amazon.nova-pro-v1:0, us-east-1) under strict accessibility instructions: exactly one factual present-tense sentence, no identity guessing, no motive speculation, no describing the camera or the timestamp.
  4. A guardrail audit that measures. Every description is then checked by code, and the result is displayed pass or fail — see below, because this is the part worth reading.
  5. Spoken caption. The sentence is spoken, and the transcript stays on screen.
  6. Loud refusal. A pitch-black or unusable frame produces a crimson LOUD REFUSAL state naming the reason — "Frame is pitch black (average luminance 0.0/255). No visual features are discernible." — not a polite fallback and not a crash.

The part worth reading: the guardrails used to be a lie

The surface printed three green ticks under every description — "✓ No Identity Speculation", "✓ No Motive/Intent Guessing", "✓ Present Tense". They were hardcoded in index.html. Nothing in the codebase ever inspected the text.

And the text they were sitting under, returned by Nova Pro from a real frame, read:

A man wearing a blue jacket and jeans stands on the porch holding a large cardboard box.

"A man" is an identity claim derived from appearance — exactly the thing the first tick certified had not happened, and exactly the thing a blind user cannot check. The tick was above it, green, in the shipped build.

So the ticks are gone, replaced by a real checker (services/descriptor/src/guardrail-audit.ts). It flags:

  • identity inference — gendered nouns and pronouns, age claims, occupation claims, named-individual guesses
  • motive inference — delivering, stealing, trying to, appears to, waiting for, about to
  • single present-tense sentence — more than one sentence, past-tense auxiliaries, future tense

Each rule names the offending token. In the demo video you can watch the audit flag our own headline description in amber while the other two rules pass. A tick nothing measures is worse than no tick at all, because it launders the failure it was supposed to catch.

Word-boundary matching is tested against the traps that make this class of code ship broken: holding, golden, shoulder and womanhood must not fire, and they do not.

How we built it

services/descriptor — Node + TypeScript. Ring webhook parsing, the watermark cropper, the luminance gate, the Bedrock client, the guardrail audit. 36 tests, tsx --test.

apps/surface — a Vite web surface that shows the whole pipeline as four visible steps, so a judge can see the frame that went in, the crop that was inferred on, the model's exact words, the audit result and the spoken caption, all at once.

Amazon Bedrock Nova Pro — real Converse calls. There is no mock in the shipped path: if Bedrock fails, the describer refuses loudly rather than falling back to canned text. So a green DESCRIBED badge on screen can only mean a real call succeeded — visible in the demo as Model: amazon.nova-pro-v1:0 · Region: us-east-1 · Latency: 5036ms.

What is real, and what is not

We would rather say this plainly than have a judge discover it.
Real The Ring schema normalisation and its tests. The watermark cropper and the 0%-in-payload assertion. The luminance gate. Every Bedrock Nova Pro call. The guardrail audit. The refusal paths. The web surface.
Real, and measured against the live Ring API An authenticated probe of the Ring Developers Playground returned 200 on six discovery endpoints, and the documented event history (GET /v1/history/devices/{id}/events) returned 200 with 7 live-view (on_demand) events. The sandbox device logs no motion or doorbell events, so there is no Ring event yet to trigger the pipeline. We had first blamed the token's scope for a 403, but that 403 came from a path we guessed. On 2026-09-23 a real frame came from Ring over WebRTC WHEP: POST .../media/streaming/whep/sessions returned 201, H264 was negotiated, ICE connected, one 1280×720 frame was decoded and the session closed with 200. Run through the pipeline, it came back as "A brown package is on the snowy steps." It comes from the Playground's sandbox device, not a customer camera. Ring's documented image download (POST .../media/image/download) returned 303, then 416: no stored image. Our earlier claim that Ring has no snapshot endpoint was wrong, because those 404s came from guessed GET paths. Evidence: docs/00-research/ring-live-api-evidence.md.
Generated, and declared on screen No frame in the demo video came from a Ring camera. The photographic fixtures are AI-generated test frames carrying signed Google C2PA content credentials (digitalSourceType: trainedAlgorithmicMedia, SynthID watermark). Every frame that uses one renders an amber strip saying so, and a provenance test fails the build if a fixture is not declared or if a scenario claims a live origin it does not have.
Honest empty state With no Playground token configured, the app shows a real 401 and says so, with a link to the console. No canned "live" responses are fabricated.

Challenges we ran into

The three green ticks. Described above. It was the most dangerous thing in the project precisely because it looked like diligence.

The first real frame was refused, and the cause was our own prompt. The Playground frame shows a parcel on snowy steps. Nova Pro refused it twice: "No discernible human presence or activity in the frame." With no Ring event behind the frame, the service had invented the sensor hint sub_type="human", and the refusal rule let "no person" count as "nothing to describe". Now no hint is invented, a real hint is marked as possibly wrong, and weather and light are not grounds for refusal. Same frame, same model: 0 of 2 described before, 6 of 6 after. A regression test pins it.

WHEP answered 500, not 4xx. A library-built offer (VP8 and one H264 profile) got HTTP 500. An offer built by headless Chrome, which lists every H264 profile, got 201 and a frame. It is in the friction log.

A 401 is more useful than a mock. The sandbox token lives 30 minutes. Rather than paper over expiry, the surface renders the genuine 401 with the console link — and a test asserts that a missing token produces TOKEN_REQUIRED, not a silent fallback.

The watermark is a content problem, not a cosmetic one. Left in the payload, the model reads the timestamp aloud. Cropping is cheap; knowing you have to is not, and nothing in the docs warns you.

Quoted UI strings drift from the UI. An earlier revision of our own submission documents quoted three on-screen strings the app did not render. They were replaced with the verbatim strings, and the lesson is in the friction log.

Accomplishments we're proud of

  • A guardrail that measures instead of asserting, and a demo that shows it flagging our own output rather than hiding it.
  • 36 service tests, including six provenance tests that fail the build if a fixture stops declaring what it is.
  • 0% of watermark pixels reach the model, asserted rather than claimed.
  • A loud refusal that a blind user can act on, instead of a fallback sentence they cannot verify.
  • Ring API claims upgraded from a single 401 to six authenticated 200s with request IDs, and the gaps written down instead of glossed.
  • A real Ring frame over WHEP, described by the same guarded pipeline. The token never leaves the capture script.

What we learned

That a verification claim is the easiest thing in software to fake, including from yourself. Three ticks, a 0%, a green badge — each of them felt like rigour and two of them were decoration. The rule we now apply across the portfolio: if a number or a tick appears on screen, something in the codebase must compute it, and a test must fail if it stops.

What's next for Doorstep

  • WHEP in the event path. Capture works: one Playground frame, 2026-09-23. Next, run it on every event, so each description comes from the camera by construction rather than from a fixture.
  • On-device TTS quality — the demo speaks through Web Speech; Amazon Polly Neural is the better voice for a doorbell announcement.
  • Multi-language description, because a doorbell announcement is useless in a language the resident does not speak.
  • Nova Pro needs an abstention path. Shown a low-information frame it invents rather than declining. We built the refusal ourselves from luminance and error states; a first-class "insufficient visual evidence" response would be safer for everyone building accessibility on this model.

Licences

Ring, Amazon and Bedrock are trademarks of Amazon. No third-party copyrighted footage appears in the demo. The narration is synthesized with Microsoft Edge Neural TTS, and the closing card says so.

Built With

Share this project:

Updates

Submission history