Custos

Custos is a self-hosted MCP server and simulation framework built for the Amazon hackathon. It ensures robust, verifiable, reliable and schema-validated agent automation.

Comment

Inspiration

Ask a voice assistant to do five things at once, and it answers "Done!" Often, it isn't.

People using Alexa+ describe the same pattern: the wrong light gets switched, a routine quietly skips a step, a reminder never fires, or something they taught the assistant is forgotten a few days later. When that happens at home, it isn't a technical problem. It's a trust problem. If you can't tell whether the coffee maker is really on, or whether tonight's dinner is safe for a child with an allergy, you stop relying on the assistant.

So we asked a simple question: what if "done" had to be proven?

What Custos does

Custos is an open-source MCP server that sits between an assistant and your home. It does three things:

  • Remembers. Household rules ("Maya is allergic to beef", quiet hours, device limits) are stored as typed, inspectable records, not as a sentence the assistant may forget.
  • Checks. Before any action runs, the rules are checked in code. Beef tacos are blocked and lentil stew is chosen.
  • Proves. After every action, Custos reads the device state back, retries once if it doesn't match, and writes a receipt. You can ask "what did you just do?" and get the evidence. If something can't be proven, it says so instead of claiming success.

In our side-by-side simulator, one request ("coffee maker and kitchen light on, coffee at 8, hall light before my daughter gets home, and dinner for us") produces "Done!" with 0 of 5 outcomes proven without Custos, and 5 of 5 backed by evidence with it. Custos also keeps scheduled and executed as separate states, so a saved schedule is never reported as a completed action.

How we built it

  • Server: TypeScript, FastMCP and Hono. Streamable HTTP, stateless, MCP spec 2025-11-25, with Bearer-key authentication. There is no language model inside the server: every decision is deterministic code.
  • 13 tools in four groups: memory, actions, confirmation and reads. Every action goes through guard, act, read back, retry once, receipt.
  • Policy engine with typed rules and an allergen ontology covering 15 allergen keys and 7 diets.
  • Adapters: a simulated device twin with injectable faults (lost acknowledgement, delay, offline, flaky), ntfy push notifications, Open-Meteo, TheMealDB and Supabase.
  • Simulator: React and Vite comparison UI, with a Strands Agents SDK agent (AWS) calling the same /mcp endpoint any other client would use. Hosted on Vercel and Render.
  • Benchmark: seeded fault injection with committed raw results.

What the benchmark shows

On a controlled harness (seed 7), Custos had 0 of 156 silent failures versus 49 of 156 (31.4%) for the baseline, and 0 of 204 rule violations versus 204 of 204 for the baseline. It disclosed all 13 unverifiable outcomes honestly.

We want to be clear about the limits. The benchmark uses a deterministic oracle, not real Alexa+. The baseline's 100% violation rate is true by construction because it has no guard. The prompt-injection sample is only three attempts.

Challenges

  • Version friction. The Strands TypeScript SDK needs Zod v4, which broke our original v3 pin. A few APIs we planned around weren't in the installed SDK.
  • Transport details. A plain JSON-RPC request fails with "Not Acceptable" unless the client accepts both JSON and event streams. The error doesn't name the missing header.
  • Windows. FastMCP refused connections on 127.0.0.1 while localhost worked.
  • Proving things we can't observe. ntfy cannot confirm delivery of a scheduled message before it fires, so Custos reports "accepted by provider" and nothing stronger.
  • Free-tier limits. OpenRouter's free quota ran out during testing, so we built a replay mode.

All of this is documented in our friction log and product feedback.

What we learned

  • Enforcing policy in code at the tool boundary is far more reliable than putting it in a prompt.
  • The most important design decision was the vocabulary of outcomes: verified, verified_after_retry, unverified, blocked, needs_confirmation. Once "I couldn't prove it" is a first-class answer, honesty becomes the default.
  • A receipt turns a vague feeling of "I don't trust it" into something a family can actually check.

Honest status

  • Devices are simulated through a device twin. No physical device or live Alexa+ session is involved.
  • Amazon documents an MCP add-on path for Alexa+. We have not validated onboarding or compatibility end to end. The same /mcp endpoint is what such an integration would call.
  • Ingredient matching uses heuristics and is not allergen-safety advice.

What's next

Validate the Alexa+ add-on path end to end. Add a real device adapter (Matter or Home Assistant). Grow the injection test set, publish model-driven benchmark runs, and add more diet packs such as halal and kosher.

Custos is MIT-licensed and open source. It is not affiliated with or endorsed by Amazon.

Built With

Share this project:

Updates

Submission history