Threshold Doorbell

A video doorbell for people who cannot hear it. Live captions of what your visitor says, and a tap-to-speak reply spoken back out the door. A full two-way conversation, no hearing required.

Comment

Inspiration

Priya is 34, profoundly deaf, and lives alone. She owns a Ring doorbell and it works exactly as designed: she gets a phone alert and sees a man at her door. Then the interaction dies. He is saying something, maybe which flat he wants, or whether to leave the parcel next door, and she has no access to any of it. Her only options are to ignore the door or open it to a stranger and mime. Her doorbell tells her someone is there. It has never once told her what they want.

We looked at the whole market for deaf and hard-of-hearing doorbells. Every product solves the same half of the problem: alerting. Strobe lights, flashing chimes, extra-loud chimes, vibrating pagers. Not one solves the conversation. The real incumbent is ambient captioning like Google Live Transcribe, but that captions the room you are standing in; it cannot tap the doorbell stream, and it is one-way, so it can never reply through the door. That gap is the whole idea.

What it does

Threshold turns a Ring doorbell into a two-way, text-first conversation device.

  1. A person-detected webhook fires. The phone plays a distinct haptic pattern (a deaf user cannot rely on a chime) and shows the visitor thumbnail.
  2. The resident taps Open. A Ring WebRTC session starts: audio both ways, video inbound.
  3. The visitor speaks. Captions stream in, large and high-contrast, styled by confidence so uncertain words look uncertain rather than being presented as fact.
  4. Three contextual smart replies appear next to a free-text field.
  5. The resident taps a reply. On-device text-to-speech turns it into audio, that audio is injected into the outbound track, and the visitor hears it spoken from the doorbell. The visitor answers, and captions continue.
  6. The whole exchange is saved as a searchable transcript of what was said at the door.

Two capture modes: Clarity Mode (Amazon Transcribe streaming, the default) and Privacy Mode (on-device Whisper, so doorstep audio never leaves the handset).

How we built it

  • Flutter app (iOS and Android), Riverpod state, built to a real design system (tokens, themed components, light and dark) rather than one-off screens.
  • Node and Fastify backend: Ring OAuth exchange and refresh, an idempotent webhook receiver (HTTP 200 within the required 5 seconds), an Amazon Transcribe streaming relay, an Amazon Bedrock smart-reply call, and conversation storage. The logger enforces a redaction allowlist so tokens, PII, and transcript text never reach the logs.
  • Ring Partner API over WebRTC and WHEP. The session negotiates audio as sendrecv, the bidirectional channel Amazon's own sample renders but never sends into. Reply audio is injected through an in-app WebView audio pipeline (ARCH-1).
  • Amazon Transcribe streaming (Clarity captions) with partial-results stabilization; Amazon Bedrock for bounded three-option smart replies; on-device Whisper for Privacy Mode.
  • Accessibility throughout: WCAG 2.2 AA, a 4.5:1 caption contrast floor, 200% text scaling, a haptic vocabulary, screen-reader support, reduced-motion.

What we learned

  • Position on two-way, never on captions. If the demo could be replaced by pointing Live Transcribe at the door, the pitch is wrong.
  • De-risk the one thing that can kill the project first. We spiked outbound audio before building any UI, and that ordering shaped the whole build.
  • A streaming relay must treat "audio before the upstream is ready" as normal, not an error.
  • Prove redaction by asserting on serialized log output, not by trusting config.
  • On Flutter, ship the cloud caption path (Transcribe) as the default and treat on-device Whisper as a bonus. Reliability wins demos.

Challenges we ran into

  • Getting audio out of the doorbell at all was the single hardest unknown. Ring's audio is sendrecv, but the canonical sample only renders the inbound half.
  • flutter_webrtc does not expose Insertable Streams, the clean way to inject a synthesised frame into an outbound track. So we moved the audio pipeline into a WebView, where the AudioContext to MediaStreamTrack path is well-trodden, and drove the outbound track from there. That decision (ARCH-1) was made early and deliberately, not patched in at the end.
  • WebSocket close-code propagation over the in-process test transport did not surface reliably within a tick, so we assert the server-initiated close path directly and poll a fixture for the disconnect path.

Built With

Share this project:

Updates

Submission history