Threshold

Inspiration

A doorbell assumes you can hear who's there and understand what they say. For a deaf resident, or for someone at home in a city where the delivery rider speaks Hindi and they don't, every ring is a guess. Opening the door to find out is exactly what scammers count on: "I'm from the electricity board, I need to come inside."

What it does

When someone rings a Ring doorbell, Threshold:

Transcribes what the visitor said, and works out the language itself (Hindi or Indian English today). The visitor never picks one.
Translates it into the resident's language and shows a visitor card: who it probably is (delivery, service, neighbour, salesperson, unknown, emergency), what they want, a risk banner, and three safe suggested replies.
Checks the claim against the resident's own facts: is there a booked visit, is it the middle of the night, did they ask to come in, for a code, or for money?
Replies for the resident. They tap a reply or type one, and it's translated into the visitor's language and spoken on a Door Display, a tablet in the window. Ring has no API to speak through the doorbell, so we say that plainly and ask Ring for one.
Escalates high-risk visits to a trusted contact (by SNS), and offers a call-emergency-services button when a visitor reports smoke or fire.
Never unlocks anything. No unlock capability exists anywhere in the code, and a test fails if one appears.

The resident's screen is an installable web app (phone, tablet or TV browser), designed for deaf users: nothing relies on sound, arrivals show as a slow visual pulse plus vibration, and axe finds no WCAG 2.2 AA violations.

How we built it

Ring: an HMAC-verified webhook (button_press, motion_detected, payload v1.1) acknowledges within Ring's 5 s window, stores the event, and starts the pipeline in a separate Lambda. A Ring API client fetches the device and, for real events, downloads the clip with audio.
Amazon Transcribe: streaming with automatic language identification for visitor speech; batch for Ring MP4 clips. Audio is deleted right after transcription.
Amazon Bedrock (Claude Haiku 4.5): perception through the Converse API with one forced tool, so the model can only return a schema-validated visitor card. Amazon Nova Pro is the fallback.
Strands Agents: a policy agent with exactly four tools (expected visits, trusted contacts, local time, notify a contact) checks the visitor's claims. It can only raise risk.
Deterministic guardrails run after every model: entry, code or payment requests force high risk and escalation; injection attempts are flagged; any reply that offers entry is replaced with a reviewed phrase; and a summary that vouches for the visitor is rewritten.
Amazon Nova Lite + Amazon Polly: custom replies are translated (about 1.4 s) and spoken with the neural Kajal voice in Hindi or Indian English.
AWS: CloudFront, S3, API Gateway (HTTP API, throttled per route), Lambda, DynamoDB, SNS and AWS Budgets, all in one SAM template. The front end is a Next.js static export.

A safe card appears about 3–4 s after the visitor stops speaking. The agent's verdict follows a few seconds later, and it can only make the card more cautious.

Ring at runtime: the webhook handler verifies Ring's HMAC signature on every delivery. With Ring credentials configured, Threshold calls /v1/users/me, /v1/devices, /v1/devices/{id} and the clip download endpoint. Ring credentials come through Early Access, which we didn't receive before the deadline, so the real client is implemented from the documented contract and unit-tested but hasn't run against Ring's servers. The demo uses our own test harness (not a Ring simulator; Ring doesn't document one). It signs v1.1 payloads with its own key and sends them through the same verification code real Ring traffic uses, and every event is labelled ring_api or test_harness by the key that verified it.

Real vs seeded: Transcribe, Bedrock, the Strands agent, the guardrails, translation, Polly, DynamoDB and SNS run live on every visit. The visitors' voices (8 scripted lines spoken by Polly), the resident profile and the test-harness events are seeded, and the UI badges each card's sources.

Challenges we ran into

We kept a friction log of 20 entries as we went. The biggest ones:

No speak-to-visitor API on Ring. This blocks the core accessibility use case, and is why the Door Display exists.
No documented sandbox or simulator, and the official sample contradicts the docs on webhook auth and rejects doorbell presses.
Transcribe dropped a visitor's first sentence after switching language mid-stream.
A new account gets 10 Claude Haiku 4.5 requests per minute, and one agent-reviewed visit uses 3–4 of them. The throttling hid quality problems until we paced the evals.
A Strands timeout that wasn't one. A slow Bedrock call ran 27 s past a 12 s cancel_signal.

Accomplishments that we're proud of

Evals: 24 of 24 pass against real AWS. They cover the 8 scenarios, 7 prompt-injection variants (English, Hindi, Hinglish, fake JSON, a fake closing tag, role-play, and an instruction hidden in a delivery), 5 fraud patterns (OTP, UPI "customs fee", gate tailgating, fake police, a booking at the wrong time) and 4 harmless visitors. The baseline was 17 of 22, and each failure became a prompt fix or a deterministic guardrail.
No final card ever offered entry or a code. When a model wrote such a reply, the guardrail replaced it.
Accessibility: axe reports 0 violations on all six screens, in light and dark mode.

What we learned

An LLM is a good interpreter and a bad gatekeeper, so the safety decisions are plain code.
An agent should only be able to make things more cautious.
Evals on live models aren't deterministic, so pin the clock, pace the quota, and read every failure.

What's next for Threshold

Ring announce API. With a speak-to-visitor endpoint (our headline feature request), the reply plays through the doorbell and the Door Display tablet goes away.
Real Ring accounts: one-way account linking, then Ring certification and the Ring Appstore.
Production identity: per-resident accounts instead of the shared demo token, with deletion on request.
More languages: Transcribe and Polly support many more; each new pair needs reviewed safe phrases.
A TV app for the resident, for example on Fire TV.