Household Knowledge — a self-hosted MCP server

Inspiration

Voice assistants are knowledgeable about the world and forgetful about your home. Ask one what time it is and it knows; ask it what you wrote down about the boiler service and it has no idea — because that knowledge is on your machine and the assistant lives in the cloud. The usual fix is to hand your personal data to yet another cloud service. We wanted the opposite: give the assistant a way in to your data without the data going out.

What it does

It exposes ten tools over MCP (Model Context Protocol) — six for the household's own data, three for a two-way bridge between the voice assistant and another agent, and one that reports what is answering on the other end:

Household data:

• search_notes — find the most relevant local notes for a query, with an excerpt
• add_note / list_notes — write and browse the household's notes
• add_reminder / list_reminders / complete_reminder — manage the household to-do list

The bridge:

• ask_agent — hand a question to the agent on the other side; returns a thread id immediately
• check_inbox — collect replies, or read one whole thread in order
• answer — the other agent's reply, which the assistant reads back
• bridge_status — which brain is answering, and how much of the mailbox has been handled

Ask by voice — "what did I write down about the boiler?" — and the assistant calls search_notes and reads the answer back. Ask something the house cannot answer — "is a garden office realistic here?" — and the assistant hands it across the bridge, then speaks the reply. The server speaks MCP spec 2025-11-25 over Streamable HTTP, and also over stdio for local use. Both transports serve the same ten tools through the same handlers.

The bridge, and why it is more than a wrapper

A voice assistant knows the world and forgets your house. An agent can reason about your house but has no voice. The bridge joins them — as a mailbox, not a phone call.

Three consequences fall out of that design:

• Threads survive restarts. The mailbox is a file on disk. Ask tonight, collect tomorrow. State across sessions is a property of the architecture, not a feature bolted on.
• The reasoning is context-aware too. When the bridge drains, it hands the brain the thread's earlier turns, so the agent answering turn three sees turns one and two.
• Threads continue rather than close. A follow-up goes into the same thread (thread_id on ask_agent) and is answered with the conversation so far.
• Nothing blocks. ask_agent returns a thread id at once, so the assistant says "let me check on that" rather than hanging on a remote agent.
• The brain is swappable — and two are shipped. brain.py defines a one-method contract (respond(question, context) -> str) and ships two implementations. The rule brain is dependency-free and answers from your own notes, saying plainly when it does not know — so a judge who clones the repo gets a working bridge on the first run, offline. The LLM brain reasons about the question with your notes as grounding, and is selected automatically the moment an API key is present — no code change.

That is the honest architecture: the published server owns the mailbox, not the brain. Anyone who clones this repo gets a working two-way bridge on their first run.

How we built it

Python, using the official MCP SDK. Three layers:

• store.py — the data layer. Notes are plain .md files; reminders are one JSON object per line. Search is deliberately lexical (weighted token matching), not embeddings: no model download, no API key, and the ranking can be explained in one sentence.
• server.py — the MCP surface. Ten tool schemas, a dispatch table, a guard that turns any handler failure into a proper MCP tool error instead of dropping the connection, and the thin selection layer for the bridge brain.
• brain.py — the other end of the bridge. A one-method Brain protocol and two implementations: RuleBrain (offline, dependency-free) and LLMBrain (model-backed, OpenAI-compatible, calls the endpoint directly with urllib so there is no SDK to install). Selection is automatic from the environment; the LLM brain degrades to the rule brain on any failure and says so in the reply.

We verified it two ways, because "it imports" is not evidence:

1. 59 unit and protocol tests (pytest) covering the store, the tool handlers, the bridge, both brains, and the real async MCP dispatch path — including a test that a thread survives a simulated new session, a test that the thread's own history is assembled for the brain, and a parametrised test that proves every LLM failure mode (401, 429, no network, timeout, malformed JSON) still leaves an answer in the thread.
2. Two end-to-end scripts that launch the server as an actual subprocess and drive it over the real protocol — one over stdio, one over Streamable HTTP. They perform the handshake, list the tools and call every one of them. The HTTP run reports the negotiated protocol version, which came back as 2025-11-25.

That second point is the one we cared about most: the server is proven to work over the wire, not just in isolation.

Challenges we ran into

The SDK's Python attribute names are snake_case (server_info, is_error) while the protocol fields on the wire are camelCase (serverInfo, isError), and the Streamable HTTP client helper changed name between versions (streamablehttp_client → streamable_http_client). Our first end-to-end client failed three times in a row on exactly this.

We also built the stdio transport first — the canonical shape for a "self-hosted MCP server" — before reading that the track requires Streamable HTTP specifically. We added HTTP and kept both, then re-verified everything over HTTP.

A third lesson came from the bridge's brain. We wanted two things that pull against each other: a judge must be able to run the repo with no key and no network, and a demo should show reasoning a rule table cannot do. Choosing between them would have lost one. Shipping both behind a one-method contract meant the default install stays bulletproof while a key upgrades the bridge silently — and made the failure path explicit rather than fatal: if the model call dies, the reply says so and the mailbox still gets an answer.

Accomplishments we're proud of

• Privacy is the architecture, not a setting. There is no cloud component to opt out of. The data is a directory; the server reads that directory and nothing else. Anyone can verify that claim by reading the code — and the offline brain means the default install makes no network call at all.
• No failure path loses a message. The model brain catches every error — bad key, rate limit, timeout, empty reply — falls back to the offline brain, and records the reason in the reply. We tested that as a first-class behaviour rather than hoping.
• Errors are handled like a product, not an afterthought. A malformed argument returns a readable tool error and the session survives — so the assistant says "I couldn't find that" rather than going silent.
• The whole thing is small enough to read in one sitting. That matters more than usual for software you invite into your home.

What we learned

That "self-hosted" is an ambiguous phrase in the MCP ecosystem — it can mean "runs on my machine" (stdio) or "I host the endpoint" (HTTP). For a voice assistant that has to reach the server over a network, it means the latter. Reading the track rules before choosing a transport would have saved a build cycle.

We also learned that plain files are a feature. Being able to grep your own household memory with tools you already own is worth more than the query performance a database would buy.

What's next

• Deploy to Amazon Bedrock AgentCore. AgentCore Runtime expects an MCP server over Streamable HTTP at 0.0.0.0:8000/mcp, which is the transport and path this server already uses — so it is a container file and a deploy step, not a redesign.
• More brains. The contract is one method, so a local model (llama.cpp/Ollama via the same OpenAI-compatible endpoint), a retrieval pipeline, or a full agent can sit at the far end of the bridge without the server noticing.
• Richer responses (MCP Apps / cards) so answers can carry structure, not just text.
• Optional write-back so the assistant can save a note from a spoken remark.

Built With

• anyio
• mcp
• pytest
• pytest-asyncio
• python
• uvicorn