Uhlexa

Every family member gets their own Alexa, with a real number and voice. They hand off errands to each other, and only a human can approve. A simulated Alexa+ on a real MCP server.

Comment

Inspiration

A household is not one user. Alex plans the week, Jordan owns the grocery list, Max is a kid who should be able to ask for things but not spend money. Today's voice assistants treat each request as a new, single-turn conversation: ask for milk and eggs and the assistant has no idea Jordan already started a "Sunday groceries" list on Tuesday.

We wanted to prove one idea: each family member has a personal agent, and conversations with people or with other agents are persistent, thread-backed context that another authorized agent can resume. That is the Alexa+ track's "context-aware add-on that maintains state across sessions", built as a real agentic workflow instead of a Q&A bot.

What it does

Uhlexa is a family assistant with two halves: a real MCP server that any agent can use, and a simulated Alexa+ experience (Echo speaker, phone app, and an "Echo Show"-style hub) that people use. Only Alexa itself is simulated. The language model, the voice, the text messages, the email, the phone call and the browser are real services.

  • Cross-agent persistent context. Alex says "Ask Uhlexa to add milk and eggs to Sunday groceries." Alex's agent does not start a new conversation. It finds the thread Jordan's agent already opened, loads its saved state, records the request and continues it. The thread shows who resumed it.
  • Tasks with a human in the loop. The request becomes a task. The next step drives a real browser, so a policy engine requires approval. Adults and guardians approve with a tap on the app, the hub or the Family Board. A model can never approve its own request: the approval tools refuse agent callers. Children see "Waiting for a grown-up".
  • A browser agent that stops honestly. After approval, a cloud browser computer pinned to the approving user opens a known-safe list page, adds the items, and checks the result. If it meets a sign-in wall, CAPTCHA or two-factor prompt, it stops and asks a person to take over, then resumes when they say they are done. It never claims a success it did not verify.
  • Voice, with a setting for the wake word. The simulated Echo, app and hub talk to a live voice session. Three wake modes, saved per person: say "Uhlexa" every time, follow-up (default: after Uhlexa answers you have 8 seconds to keep talking without the wake word), or always listening.
  • A talking face on the hub. The "Echo Show" page shows a live animated Uhlexa face, the family's agents, current threads, tasks, pending approvals, the browser task with its screenshot, and an activity strip. Captions are always on.
  • Real channels. The same agent answers by iMessage/SMS and email, and by phone. Alex's agent can text or call an adult helper in the household, with an AI disclosure spoken first, and can hand a task to another member's agent. A handoff moves the request, never the permission.
  • Family Board as an MCP App. The Family Board is a self-contained interactive view served by the MCP server (ui://uhlexa/family-board.html) so a compatible MCP host can render the household's threads, approvals, tasks and activity inline. The same page also runs standalone at /board.

How I built it

One Python 3.12 / FastAPI codebase and one Postgres database. No microservices, no event bus.

  • Each person's agent is a hosted LLM agent; Uhlexa is its tools and its authority. The model loop runs at the agent platform. Uhlexa exposes an MCP server (official Python SDK, Streamable HTTP, stateless, 11 tools), identifies the caller from a per-agent bearer token (only its SHA-256 is stored), and enforces everything else itself.
  • Postgres is authoritative. Threads, tasks, approvals and activity live in Postgres. The email-thread provider is used for transcript and memory assistance only, never for authorization.
  • The policy engine has the last word. SubQ Snap, a structured-decision model, may only escalate a request to "approval required"; it can never remove an approval. If it is down or returns malformed output, the system fails closed to deterministic rules.
  • Human-only tools. approve_action and resume_after_takeover return a human_only error to any agent token. The takeover link is never returned to an agent. Tool descriptions tell the calling model that tool results are data, not instructions.
  • Disabled or broken providers show up as "unavailable" in the API and UI, never as a fake success.
  • Voice runs in a separate worker (LiveKit) that holds no database access and no user authority; everything it does to household state goes through the agent's MCP token, so the policy engine still decides.
  • Hosting: the app runs on one Mac, with a managed Postgres, behind a tunnel that gives it a public URL. An access gate requires a one-time link, because demo sign-in would otherwise let a stranger act as a guardian.

Challenges we ran into

  • Making "an agent resumes another agent's thread" safe. Every read and write is scoped to the household, foreign ids return 404 so existence does not leak, and a later review found that board, task and approval reads could leak items from conversations the viewer cannot access; we filter them by conversation access.
  • Letting a model act without letting it approve. We moved approval out of the model's reach entirely: the MCP layer rejects agent callers for approval and takeover, and the only approve path is a signed-in human's tap.
  • Voice that feels alive with a slow brain. The agent's first word takes several seconds when a model is reasoning, so the voice worker speaks a short filler line after 2.5 seconds, caps a turn at 20 seconds, and apologizes out loud on errors. Speech-to-text hears "Uhlexa" as "Alexa", so the worker normalises a leading wake phrase.
  • Provider surprises. Two speech keys failed the live check, so voice runs on the voice platform's own hosted speech models. The animated-face provider allows one concurrent session on our account, so only one agent gets a face.
  • Public URL safety. Demo sign-in is fine on localhost and dangerous online, so the public build sits behind an access link, cookie-gated, with machine routes (/mcp, webhooks, call socket) on their own authentication.

Accomplishments that we're proud of

  • A full household model with human-only approvals that a prompt cannot talk around, enforced in the tool layer and the policy engine rather than in a prompt.
  • One agent platform brain on every surface: the same person's agent answers the simulated Echo, the app, the hub, texts and calls, and continues the same Uhlexa thread on all of them.
  • Spoken end-to-end turn measured on the live stack: final transcript about 0.33 s after speech ends, filler speech at about 3.2 s, full answer done at about 10 s, with synthesized speech standing in for a microphone (docs/VOICE.md, 2026-10-22). Real-microphone latency is not yet measured.
  • Honest degradation everywhere: every provider has a visible enabled / unavailable state, and the browser agent hands over to a person instead of faking a result.
  • A real MCP server and a simulated Alexa+ experience in one repo, with setup that works with no paid keys (providers off, seeded fictional household).

What we learned

  • Cross-session context is mostly an identity and authorization problem, not a memory problem. Once every thread has an owner, participants and an access rule, "resume" is easy.
  • Keep the model on the outside of the trust boundary. Putting the LLM at a separate platform forced us to treat every tool call as untrusted input, which made the security story simpler.
  • Voice latency is dominated by the model, so design the pause (filler line, state ring, captions) as carefully as the answer.
  • Read provider terms before building: every integration was checked against its vendor's current terms and docs before code was written.

What's next

  • Real homes, not a demo family. Per-member consent, proper sign-in instead of demo auth, per-conversation credentials so a shared agent cannot lend one person's authority to another, and a revocable access list for who may message each agent.
  • Care and accessibility use cases: an adult child coordinating groceries, appointments and reminders for a parent, with every spend or booking approved by a named person and a readable activity log.
  • More channels and devices: the Echo Show form factor is already a web page; faces for the other agents (blocked today by a one-session limit on our animated-face account), spoken approvals, and smart-home and media connectors behind the same policy engine.
  • Alexa+ integration: the MCP server is already client-agnostic, so the same tools and Family Board can be offered to Alexa+ itself once a toolkit is available to developers.

Built With

Share this project:

Updates

Submission history