Accessible door intelligence

Inspiration

Meet Mai. She is 72, lives alone, and is hard of hearing. When the doorbell rings while she is cooking, the chime means nothing to her, and a phone thumbnail means little more. The TV in front of her could say who is at the door, in plain text, in seconds — but today it mostly does not.

That gap is also a safety gap. Anyone can stand at a door and say, "I am the police, open the door." The FTC's Consumer Sentinel Data Book 2024 counts 845,806 imposter-scam reports and $2.95 billion in reported losses, with government-impersonation losses alone at $789 million, and adults aged 70 and over reporting the highest median losses of any age group. Mai is a composite of the residents this is built for, not a real person's data, but the risk she faces is measured, not invented.

Existing tools transcribe the door. Ring Video Descriptions turns camera events into text, and Alexa+ Greetings can handle a delivery. Neither adjudicates the visitor's claim. AccessBell was built for that missing step: not a new camera, and not caregiver surveillance, but a door that explains itself in text and checks what it is told before it repeats it.

What it does

AccessBell turns a Ring door event into accessible text on the screens a household already owns, and it checks the visitor's claim instead of repeating it.

The flow, in order, is:

• A Ring event arrives. Tier 1 rules instantly show "Someone is at the door" on the TV-first surface. The caption never waits for a model.
• Tier 2 then produces a grounded caption in English and Vietnamese, with its evidence shown beside it. If a specific category has no grounding evidence, it is demoted in code to a generic caption. The model never gets to invent a visitor.
• The Gatekeeper then checks the claim itself. A claim such as police, pharmacy, utility, government, package, or family is verified only against resident-created context — an expected window or a calendar event the resident controls. It comes out as verified, unverified, or contradiction, with a risk level and a claim-specific challenge question, for example "Ask for order number / ID and callback number. Do not open." The visitor's own words, a previous user label, and an unanalysed snapshot are never treated as proof.
• The resident can act from where she already is: large on-screen buttons on the TV, a quick reply through Alexa+ MCP tools, or a short talk at the door. Memory suggests the same reply as last time, but only for the same event category at the same door, and only as a one-tap suggestion — never automatically.
• Sharing and escalation are a safety net, not a default. A share brief stays locked until the resident ticks consent. Escalation is off by default, durable across restarts, at most once, revocable, and written to an audit log. There is no face recognition, no face embeddings, no video archive, and events are purged after 30 days.

In the demo, the same police claim is first unverified, then verified only after the resident creates a matching police patrol window. Repetition does not create trust. A resident-created schedule does.

How we built it

AccessBell is a FastAPI backend with a TV-first PWA web UI, a self-hosted MCP server, and an AWS CDK deployment path. The runtime path is:

Ring Partner API webhook v1.1 (HMAC-SHA256, X-Signature, dedupe by request_id, inside Ring's 5-second delivery budget) → Tier 1 rules → Tier 2 grounded triage → grounding gate → Gatekeeper verification against resident windows and calendar → captions over SSE on the PWA / Fire TV surface, Alexa+ via MCP, and consent-gated share and escalation.

How it works, technically:

• Ring: the demo uses an in-app Ring simulator that builds a real v1.1 envelope, signs it with HMAC-SHA256, verifies the signature, and passes it to the same RingWebhookHandler used by the real webhook route. The simulator is disclosed as a simulator. One demo field, visitor_statement, is an AccessBell transcript extension, not a standard Ring field, and the README says so plainly.
• Triage: Tier 2 uses LangChain structured output with a pydantic TriageResult contract. In the AWS path the model is Amazon Bedrock, by default us.amazon.nova-2-lite-v1:0. In local tests and in the recorded video the provider is a deterministic stub with the same contract, so tests and demos never depend on network or credentials. LLM output is re-validated in code and never reaches a user unvalidated.
• Alexa+ MCP: a self-hosted MCP server over Streamable HTTP, JSON-RPC 2.0, MCP spec 2025-11-25, at POST /mcp, with exactly 11 tools: get_door_alerts, get_event_detail, get_context_snapshot, confirm_event, send_quick_reply, add_expected_context, list_expected_context, clear_expected_context, update_accessibility_prefs, set_escalation, and get_daily_summary. The MCP server is protected by a Cognito OAuth 2.1 resource server. The video shows the real tools/list and tools/call trace through an in-app Alexa+ MCP client simulator, not a physical Alexa+ device, and the submission says so.
• AWS Builder: Amazon Bedrock for triage, Amazon EventBridge as an optional outbound bus (off by default), Amazon Cognito for MCP OAuth 2.1, Amazon ECS Fargate with an Application Load Balancer, Amazon RDS Postgres, Amazon SNS / SES for escalation and announce transport, and Secrets Manager for generated tokens. The whole stack deploys with ./scripts/deploy_aws.sh. Storage is portable SQLite or Postgres, and policy.yaml carries the version and hash cited on every audited row.
• Accessibility, built as a contract, not an afterthought: English and Vietnamese captions from a single source in captions.py, kept short and TTS-safe so a screen reader and a voice line read the same thing; D-pad navigation, 44 px targets, a 10-foot TV presentation, child mode under the same consent gate, and quiet hours that change phrasing only, never evidence.
• Evidence, reproducible from the repo: pytest (427 collected: 390 pass, 37 Postgres-skipped without a Postgres DSN), a 46-case triage and Gatekeeper eval, a grounding ablation, a latency measurement, and an MCP verification script that boots the app and runs initialize, tools/list, and tools/call.

Challenges we ran into

• The hardest problem was not transcription, it was adjudication. A friendly model will happily repeat "a police officer is at the door" as if it were a fact. We solved that by making verification a code gate, not a prompt instruction: the Gatekeeper checks only resident-created windows and calendar events, and an unverified high-stakes claim keeps its warning headline instead of being softened into a friendly caption. Prompt guardrails can be ignored by a model. A code gate cannot.
• The Alexa+ MCP client does not yet support the WWW-Authenticate discovery header, while standard OAuth expects that header. Following the standard literally would therefore break the Alexa+ client. We added an explicit flag, MCP_401_WWW_AUTHENTICATE=false, documented the deviation in the friction log, and kept the standard behaviour as the default. That is a real integration lesson, not a workaround hidden in code.
• We had no physical Ring doorbell, Fire TV device, or Alexa+ device. Instead of pretending otherwise, we built a signed Ring simulator on the real webhook path and a TV-first PWA that is D-pad navigable, and we state both limits in the README, in the submission notes, and on the honesty slide of the video. Fire TV is a surface in this entry, not the submitted track, and the UI is a web app, not a packaged Fire TV app.
• A door caption must never wait for a model, and a model must never be the only check. Tier 1 rules therefore run first, with a 200 ms context budget and a 2,000 ms enrichment budget, failing open to a generic caption when enrichment is slow or missing. That is why the local Tier 1 path is fast, and why we qualify every latency number as a local path, not Ring-cloud-to-TV end to end.
• Bilingual captions turned out to be a design problem, not a translation problem. Vietnamese copy had to be short, diacritic-safe, and carry exactly the same meaning as the English line when read aloud. We reviewed it with automated caption tests and Vietnamese copy review. We do not claim a certified human translation or a full screen-reader audit, and the accessibility statement says so.

Accomplishments that we're proud of

• The Gatekeeper, with every case matching its expected outcome in the 46-case eval: 46 out of 46 cases pass, covering category, Gatekeeper outcome, memory, demotion, and escalation.
• Grounding that can be measured, not asserted: in the grounding ablation, the grounding score rises from 0.5 without context to 1.0 with expected context on the labelled set, because the gate demotes ungrounded claims in code.
• A fast local path that does not hide behind a model: Tier 1 rules at 0.1 ms at p95 locally, and webhook to caption at 14.5 ms at p95 on the local path. Both numbers are local-path numbers with the stub provider, and we label them as such.
• A consent safety net that stays a safety net: escalation is off by default, durable, at most once, revocable, and audited, and sharing stays locked until the resident consents. The resident, not a watcher, is the user.
• An MCP surface that actually runs: 11 tools, verified by a script that boots the app and calls initialize, tools/list, and tools/call, with a daily brief in Vietnamese and a reply path that works from the same tools Alexa+ would call.
• A complete, checkable submission: a public MIT-licensed repository, a reproducible evidence suite, product feedback for every API and SDK used, and a friction log in the format Amazon asks for. Nothing in the demo is claimed as hardware that was not used.

What we learned

• Transcription, grounding, and verification are three different jobs. A caption can be grounded in an event and still carry an unverified claim, and a claim can be verified against a schedule while its caption stays generic. Designing them as separate badges, with separate definitions, was the difference between a doorbell that talks and a doorbell that can be trusted.
• A prompt is not an enforcement mechanism. If a safety property matters, it has to live in code, in a validated contract, and in a test that fails when it breaks. That lesson now shapes every outward action in the system: only share and escalation leave the house, both are opt-in, and both are audited.
• Accessibility has to be a contract first. Caption length, TTS safety, D-pad focus, target size, and a single English and Vietnamese source are not polish added at the end. When they are the contract, the same door works for hearing, vision, mobility, language, and age, without a separate version for each.
• Honesty is a feature, not a disclaimer. Saying plainly that the Ring input is a simulator on the real webhook contract, that the recorded triage model is a stub, that visitor_statement is an extension, and that Fire TV is a surface rather than a packaged app, lets anyone test what is actually there, without being asked to believe what is not.
• Consent has to be boring to be safe. Off by default, one tap to reuse a previous reply, revocable at any time, and an audit log the resident can keep as a plain-text door log. A safety net that is always on is not a safety net. It is surveillance.

What's next

From the repository roadmap, in order, not as a wish list:

• Live Ring sandbox end to end, so the signed simulator path is validated against a real Ring callback, and a packaged Fire TV app, so the TV surface stops being a PWA layout and becomes a Fire TV app in its own right.
• Richer claim taxonomy and resident-defined verification rules, so a household can add its own expected patterns without changing code, while keeping the rule that only resident-created context can verify a claim.
• A public HTTPS deployment with Postgres through the existing CDK stack, including a live Postgres pass for the door-turns table, which is currently tested on SQLite, and a full screen-reader audit plus certified Vietnamese translation review.
• A real Alexa+ connection over OAuth 2.1 with PKCE, replacing the in-app MCP client simulator, and a one-frame snapshot inside a consented share only — never as an analysed identity proof, and never without consent.
• Richer voice history with event citations, and speak-at-door replies where the available APIs allow them, so a resident can answer in full sentences without giving up the fixed-phrase honesty guarantee for translation.

Built With

• amazon-web-services
• bedrock
• ecs
• eventbridge
• python
• sns