HomeKeeper

Your home's memory, on Alexa+. An MCP server that remembers every appliance, reads the manuals, plans maintenance, and reorders the parts you forget.

Comment

🚀 Try it live: https://d157gr6uv21vdd.cloudfront.net (the simulated Alexa+ host on AWS, talking to the HomeKeeper MCP server on Bedrock AgentCore Runtime. Type a question or press Talk for live voice. Everyone shares one demo household.)

🔗 Code: https://github.com/ritwikareddykancharla/homekeeper-alexa (MIT)

💡 Inspiration

Every household owns 20 to 40 appliances, each with a manual nobody keeps, a warranty nobody tracks, and a filter nobody remembers to replace. The information exists. It is just scattered across PDFs, receipts, and the back of a cabinet. A voice assistant that lives in the kitchen is the natural place for it, but only if it can actually remember across sessions and read the source material instead of guessing.

🗣️ "Alexa, the dishwasher is showing E24." 🔊 "That's a drain error on your Bosch 300 Series. Check the drain hose for a kink and clean the filter under the lower rack. Want me to show you the steps?"

🎯 What it does

HomeKeeper is an Alexa+ add-on built as a self-hosted MCP server (spec 2025-11-25, Streamable HTTP). It gives Alexa+ four abilities it does not have today:

  • 🧠 Remembers. Appliances, purchase dates, warranties, and maintenance history persist per household across every conversation.
  • 📖 Reads. Manuals are chunked, embedded with Amazon Bedrock, and retrieved per question, so troubleshooting answers are grounded in the actual document, with the source excerpt shown as a card.
  • 📅 Plans. When you register an appliance, HomeKeeper derives its maintenance schedule (filters, descaling, inspections) and surfaces what is due.
  • 🛒 Acts. Consumables are linked to products, so "order the fridge filter" is one turn, confirmed through MCP elicitation.

🧰 Tools exposed

Tool What it does
register_appliance Add an appliance (brand, model, room, purchase date). Auto-derives its maintenance schedule.
list_appliances Household inventory, rendered as a card.
ingest_manual Attach a manual (URL or upload) and index it for retrieval.
troubleshoot Grounded Q&A against the appliance's manual, with a source-excerpt card.
maintenance_due What needs attention now and in the next 30 days.
log_maintenance Record a completed task and reset its schedule.
reorder_consumable Find and order the matching filter, part, or consumable.
warranty_status Is it still covered, and what is needed to claim.
maintenance_history What has been done, and when.
remove_appliance Forget an appliance.

Plus a weekly_checkup prompt and four ui://homekeeper/*.html MCP App resources.

🏗️ Architecture

System overview

                         ┌───────────────────────────────────────────────┐
                         │               🏠  HOUSEHOLD                   │
                         │   Echo / Alexa+ app  ·  or the web simulator   │
                         └───────────────┬───────────────────────────────┘
                                         │  voice / text
                                         ▼
┌────────────────────────────────────────────────────────────────────────────┐
│  🗣️  MCP HOST  (Alexa+, or packages/simulator standing in for it)          │
│                                                                            │
│   LLM reasoning ──► tool selection ──► MCP client ──► renders MCP App cards │
└───────────────────────────────┬────────────────────────────────────────────┘
                                │  Streamable HTTP  ·  JSON-RPC  ·  session id
                                ▼
┌────────────────────────────────────────────────────────────────────────────┐
│  ⚙️  HOMEKEEPER MCP SERVER      (packages/server, TypeScript, Node 22)       │
│  hosted on Amazon Bedrock AgentCore Runtime                                 │
│                                                                            │
│   ┌──────────────┐   ┌──────────────┐   ┌──────────────┐   ┌────────────┐  │
│   │ 10 tools     │   │ 4 UI views   │   │ 1 prompt     │   │ elicitation│  │
│   │ (mcp.ts)     │   │ ui://…html   │   │ weekly_checkup│  │ (orders)   │  │
│   └──────┬───────┘   └──────────────┘   └──────────────┘   └────────────┘  │
│          │                                                                  │
│   ┌──────▼─────────────┐  ┌─────────────────────┐  ┌─────────────────────┐ │
│   │ service.ts         │  │ knowledge/          │  │ ai/bedrock.ts       │ │
│   │ schedules, orders, │  │ retrieval.ts        │  │ Claude Sonnet 4.5   │ │
│   │ warranty logic     │  │ manuals, catalog    │  │ Titan Embeddings V2 │ │
│   └──────┬─────────────┘  └──────────┬──────────┘  └──────────┬──────────┘ │
└──────────┼───────────────────────────┼────────────────────────┼────────────┘
           ▼                           ▼                        ▼
   🗄️  Amazon DynamoDB          🪣  Amazon S3             🤖  Amazon Bedrock
   household state              manual chunks +          Converse API +
   (single table)               embeddings               embeddings

🔄 One turn, end to end: "the dishwasher is showing E24"

  user ──1──► host LLM ──2──► troubleshoot(appliance="dishwasher", question="E24")
                                      │
                                      ▼
                    ┌─────────── HomeKeeper server ───────────┐
                    │ 3  resolve appliance  (DynamoDB)        │
                    │ 4  load manual chunks (S3)              │
                    │ 5  hybrid retrieval:                    │
                    │      cosine(Titan embeddings)           │
                    │    + BM25 keywords                      │
                    │    + exact error-code boost  ("E24")    │
                    │ 6  Claude synthesises from chunks ONLY  │
                    │ 7  return text + _meta.ui.resourceUri   │
                    └─────────────────┬───────────────────────┘
                                      │
                 ◄──8──  spoken answer ┴ troubleshoot card (manual excerpt, steps)

🎙️ Live voice path (simulator)

  🎤 mic ─── PCM ───►  Amazon Nova 2 Sonic  (one bidirectional stream)
                            │
                            ├─ speech recognition
                            ├─ reasoning + tool selection
                            ├─ toolUse ──► MCP client ──► HomeKeeper ──► result ──┐
                            │                                                     │
                            ◄─────────────────── toolResult ◄─────────────────────┘
                            │
                            ├─ text out ──► Amazon Polly (Joanna, the classic Alexa voice) ──► 🔊
                            └─ barge-in: user speaks, playback stops, stream continues

Typed turns take a parallel path through the Bedrock Converse tool loop and are spoken by the same Polly voice. If Bedrock is unreachable, a rule-based intent router keeps every tool and card working.

🛒 Order confirmation: elicitation with a fallback

  reorder_consumable(appliance, consumable)
            │
            ├─ host supports elicitation? ── yes ──► server.elicitInput(confirm?) ──► order placed
            │
            └─ no ──► return quote ──► host asks user ──► reorder_consumable(..., confirm=true)

Ambiguous references ("the fridge" when there are two) return candidates instead of guessing.

☁️ Deployment (AWS CDK, infra/)

   npm run deploy
        │
        ├──► HomeKeeperStack
        │    Bedrock AgentCore Runtime (MCP protocol mode, Node 22 direct-code zip, session affinity)
        │    DynamoDB table  ·  S3 bucket  ·  IAM role  ·  Cognito user pool (optional, -c auth=jwt)
        │         │
        │         ▼
        │    https://bedrock-agentcore.us-east-1.amazonaws.com/runtimes/<arn>/invocations
        │         ├─ IAM (SigV4)  ── default
        │         └─ OAuth JWT    ── what Alexa+ account linking expects
        │
        └──► HomeKeeperDemoStack  (the public link above)
             EC2 host runs the simulated Alexa+ host from the GitHub main branch
             CloudFront in front for HTTPS (microphone) and the voice WebSocket
             instance role: Bedrock, Polly, AgentCore invoke

🚀 How we built it

  • MCP server: TypeScript on @modelcontextprotocol/sdk, MCP spec 2025-11-25, Streamable HTTP transport. Runs stateful (sessions, needed for elicitation) or stateless, and honours platform-injected session ids so it runs unchanged locally, under the MCP Inspector, and on AgentCore Runtime.
  • MCP Apps: tools declare _meta.ui.resourceUri so Alexa+ (and any MCP Apps host) renders rich cards: appliance inventory, troubleshooting with the manual excerpt, maintenance timeline, order confirmation. Views are single-file HTML served as ui://homekeeper/*.html resources.
  • Grounded troubleshooting: manuals are chunked by section and embedded with Titan Text Embeddings V2. Retrieval blends cosine similarity with BM25-style keyword scores and an exact error-code boost, so E24 always surfaces. Claude Sonnet 4.5 synthesises the answer from the retrieved chunks only.
  • Simulated Alexa+ host: designed as the screen of a kitchen display, with an ambient light field, frosted glass cards and Alexa's light bar reacting to her state. It speaks to the same server exactly as Alexa+ would and is hosted publicly at the link above. This is the demo surface while the Alexa+ MCP Toolkit is in Private Preview.

☁️ AWS services used (AWS Builder mini challenge)

Service Role
🤖 Amazon Bedrock (Claude Sonnet 4.5, Converse API) Maintenance schedule refinement, grounded troubleshooting synthesis, host-side reasoning in the simulator
🧬 Amazon Bedrock (Titan Text Embeddings V2) Manual chunk embeddings for retrieval
🏃 Amazon Bedrock AgentCore Runtime Hosting the MCP server in MCP protocol mode with session affinity
🎙️ Amazon Bedrock (Nova 2 Sonic) Live voice: speech recognition, reasoning, MCP tool calling and barge-in in one stream
🔊 Amazon Polly (Joanna, the classic Alexa voice) The voice the user hears, with SSML for rhythm and error codes; switchable live
🗄️ Amazon DynamoDB Per-household appliances, schedules, history, orders (single-table)
🪣 Amazon S3 Manual chunks with embeddings
🔐 Amazon Cognito (optional) JWT issuer for OAuth-protected inbound auth
🖥️ Amazon EC2 and CloudFront The public demo host for the simulated Alexa+ experience
🧱 AWS CDK Infrastructure as code, both stacks

🧗 Challenges we ran into

  • Making elicitation work across hosts that may or may not support it, without the LLM ever placing an order unconfirmed.
  • Keeping retrieval honest: pure embedding search misses exact error codes, so retrieval had to become a hybrid.
  • Running one server binary unchanged across local, Inspector, and AgentCore, where the platform injects its own session id.
  • Nova 2 Sonic's built-in voices were the weak link, so the stream's audio is discarded and each sentence is re-spoken by Polly Joanna.

🏆 Accomplishments that we're proud of

  • ✅ A complete end-to-end you can try: utterance to tool call to grounded answer to rendered card, including the elicitation round-trip, running on AWS.
  • ✅ Live voice with Nova 2 Sonic driving real MCP tools, with barge-in.
  • ✅ Graceful degradation: without Bedrock credentials every tool and card still works in rule-based mode.

📚 What we learned

MCP Apps plus elicitation is enough to build a genuinely multimodal assistant experience without any proprietary SDK. Grounding answers in retrieved manual text is what turns a chatbot into something you would trust with a dishwasher error code.

🔭 What's next for HomeKeeper

  • Register the hosted server with Alexa+ through the MCP Toolkit (alexa-ai new mcp) once Private Preview access lands.
  • 📸 Photo-based appliance registration (snap the model plate).
  • 👵 Shared households and caretaker access for ageing relatives.

🔓 Open Source

MIT licensed. Source, setup, and deployment instructions: https://github.com/ritwikareddykancharla/homekeeper-alexa

Built With

Share this project:

Updates