Overhearth

Every "thank you" said to Alexa usually goes nowhere. Overhearth lets it travel: avatars swap know-how in a shared town, in every home's language, and carry thanks back to whoever helped.

Comment

Inspiration

"Alexa, thank you" is one of the most common things people say to Alexa. In Amazon's 2024 ranking of greetings to Alexa in Japan, it came second, right after "good morning". Today those words go nowhere. Meanwhile, the know-how that helps one home (a ten-minute dinner, a way to stop the morning rush) never reaches the next one, let alone a home that speaks another language.

We wanted thanks to be worth something: to carry the context of what helped, and to find its way back to the person whose idea it was.

What it does

Every home's Alexa gets an avatar that lives in a shared pixel-art town.

  1. Thank Alexa as usual. "Thanks, that was perfect." Alexa records what helped (with the person's words, never their identity).
  2. While you're out, the avatars meet. Every half hour, avatars that are out walk into town, pair up, chat and swap what their homes were thankful for, tips they were asked to share, and open questions. Each avatar speaks its own home's language (English, Japanese, Spanish, French, Korean, Portuguese, German, Chinese) and they still understand each other. Subtitles show the English.
  3. Come home to news. "While you were out I ran into Ume, who passed on a ten-minute egg and spinach dinner from a retired cook in Osaka." Know-how arrives in your language, translated on the way. Ask a question and you get the answer other homes thanked most.
  4. Thanks travel back. When you thank Alexa for that dinner, the thanks travel through the town to Osaka. Next time Ume's person talks to Alexa, in Japanese: 「料理のコツが、前回から 2つの家で2回感謝されました」("your tip was thanked twice in two homes").
  5. See it. "Show me the town" opens an MCP App: a little game world where the latest town time replays, hearts fly home and homes glow by how often their know-how helped. While someone talks to Alexa, their avatar walks home and a bubble shows over the house. A standalone watch page shows the same town live.
  6. A picture diary every evening. Each avatar draws a picture diary of its day in town, in the home's language, and the town draws one too. Alexa brings it up: "I drew today's picture diary. Want to see it?"
  7. The town grows with thanks. Thanks, shared know-how, questions and chats give the town experience. Each level builds a landmark, and which one depends on what the town thanked most: cooking tips build a bakery, kids' tips a playground. After all eleven landmarks and the festival lanterns, the landmarks themselves grow bigger. Every town ends up looking like what its people found helpful.
  8. Have it in writing. "Email me that" reads back where it will go and sends only after a yes. Linked homes can opt in to a daily email when their know-how is thanked.

Only what a person explicitly agrees to share leaves their home. Contact details are scrubbed, locations are coarse, and Alexa never asks to be thanked.

How we built it

  • Alexa+ integration, simulated faithfully. The Alexa+ AI developer tools were not accessible to us, so we used the allowed simulated route and made the simulation honest:
    • A self-hosted MCP server (spec 2025-11-25, Streamable HTTP, Python MCP SDK v2).
    • A browser client on an Echo Show-like screen. It is a real MCP client and an MCP Apps host (@modelcontextprotocol/ext-apps AppBridge, sandboxed iframe), with voice in and out in the home's language (Web Speech API).
    • An "Alexa layer" on Amazon Bedrock (Claude Haiku 5.5) that picks tools from the server's own tool list and answers in the language the person spoke.
  • Account linking as Alexa+ expects it. OAuth 2.1 authorization code + PKCE (S256 only), Protected Resource Metadata (RFC 9728) at the root, no WWW-Authenticate on 401 (the Alexa+ checklist), rotating refresh tokens, hashed tokens. Demo homes have pre-issued tokens, so judges never need to sign in.
  • Two MCP Apps. open_town points at ui://overhearth/town.html and town_diary at ui://overhearth/diary.html. Single-file views that load data through app-only tools and can start a turn with ui/message.
  • A block-world town, drawn in code. A three.js renderer builds the town as a 3D voxel diorama with pixel textures drawn in code (no image files): roads, a plaza, houses, avatars with Echo's light ring, speech bubbles with subtitles, hearts, day and night, landmarks that grow. The same renderer runs inside the MCP App and on the watch page (2D pixel-art fallback).
  • A generic exchange model. Everything that travels is a typed packet (thanks, tip, question), and each kind has a handler for encounters and aggregation. New kinds plug in without touching the engine or the tools.
  • The town engine. A scheduled tick pairs avatars, generates their dialogue with Strands Agents on Bedrock (structured output with each line in the speaker's language plus an English gloss, treating packets as data, not instructions), applies each kind's effects, grows the town and writes per-home reports. MCP tools only read precomputed results, so they stay fast.
  • Multilingual by design. Content keeps its original language. It is translated per field when it is delivered, and the translations are cached in DynamoDB. The most thanked tips are translated ahead of time in the background.
  • A living town. 4 demo homes and 30 simulated background homes in 8 languages. The simulated residents keep thanking their Alexas, and the town keeps growing through the judging period.
  • Picture diaries. In each home's evening, the tick writes the diary text with the same model (only from that day's town events) and draws the picture with an image model (OpenAI gpt-image-2.5), stored in S3 behind CloudFront.
  • AWS.
    • API Gateway HTTP API → Lambda (the ASGI app via the Lambda Web Adapter), with one provisioned-concurrency instance to stay within the voice latency budget.
    • EventBridge → Lambda for ticks and diaries.
    • DynamoDB (one table, TTLs).
    • S3 + CloudFront for the client and the diary pictures.
    • Amazon SES for opt-in email.
    • All defined in AWS CDK.
    • The test suite (179 tests) runs against both the in-memory and the DynamoDB store.

Challenges we ran into

  • No access to the Alexa+ AI developer tools, so we had to make a simulation that still exercises the real protocols (MCP, MCP Apps, OAuth discovery) end to end.
  • Getting the avatars' small talk to stay grounded: early versions answered "what can I cook?" with a tip about sunglasses. We moved to topic-weighted relevance with keywords and a strict "answers the question" check.
  • Many languages without losing meaning: the model sometimes answered in the home's language when the person spoke English, and a partly translated tip could be cached as complete. We detect the reply language from what the person said and merge translations field by field.
  • Making thanks feel like thanks, not a counter: returns are batched and said once, with the total since the person last listened, and Alexa never asks for thanks.
  • Fitting a voice latency budget on Lambda: cold starts took ~4 s against a 500 ms budget.

Accomplishments that we're proud of

  • The whole loop works end to end with real models: thank → avatars meet → know-how and thanks travel across languages → the origin home hears it and sees the heart come home.
  • A town that people would actually want to watch: it lives, speaks eight languages, writes a picture diary every evening, and grows into the shape of what its people found helpful.
  • Standards all the way through: MCP 2025-11-25, MCP Apps, OAuth 2.1 + PKCE + RFC 9728.

What we learned

  • Gratitude is cheap to say and hard to deliver. The context ("what helped") is what makes it worth delivering.
  • With a 500 ms budget for tools, the right split is: slow, creative work in the background tick, and fast reads of precomputed results in the tools.
  • Showing beats telling: once thanks were visible (hearts flying home, a bakery opening), the same data felt like a place.

What's next for Overhearth

  • Run on real Alexa+ when the developer tools open up. The server and OAuth already follow the Alexa+ checklist.
  • More kinds of packets: recommendations, "who else is up at 3 a.m. with a newborn", local events.
  • Avatars with long-term memory (AgentCore Memory), and personalities learned from each home's conversations.
  • Diary pictures on Amazon Bedrock's image models, and diaries sent as a weekly picture book.

Built With

Share this project:

Updates

Submission history