HoldDesk

A voice front desk where the assistant can hold, ask and explain, while only a human confirms. Seven MCP tools, two-tier OAuth and a staff approval desk, with an Alexa-style simulator.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for HoldDesk

Inspiration

Small professional-service offices (insurance agencies, accounting firms, clinics) get their busiest calls when nobody can pick up. A voice assistant could answer them, but offices will not let an AI confirm appointments, cancel meetings or quote prices on its own. The customer is left either with voicemail or with an assistant that says "you're all set" when nobody has actually agreed. We wanted a voice assistant that is useful and accountable.

What it does

HoldDesk is a self-hosted MCP server that gives Alexa+ (or any MCP client) seven front-desk tools: check availability, place a booking hold, check status, request a cancellation, answer office FAQs from a bundled knowledge file, give a clearly non-binding home-insurance price range, and escalate to a person. The staff queue is deliberately not exposed as an MCP tool.

The product principle is structural, not a prompt: no MCP tool can approve anything. Every consequential action lands in a human approval queue on a separate staff-only /desk page, where a named person approves or rejects it and the decision is logged. The assistant is told this in every tool description, and the simulator only says "confirmed" when the status tool returns confirmed.

Because Alexa+ developer-preview access is invite-only, we also built the rules-allowed simulated Alexa+ experience: an Echo-Show-style web app with voice input (Web Speech API), spoken replies, visual cards and a live developer trace that shows every MCP call, its latency and the OAuth tier used.

How we built it

  • MCP server: Python MCP SDK (FastMCP) over Streamable HTTP, negotiating protocol 2025-11-25; every tool returns a spoken_summary for voice plus structuredContent for screens; DNS-rebinding protection. The MCP HTTP transport is stateless; demo appointments and OAuth sessions are held in one server process, with optional appointment JSON storage.
  • Auth in the OAuth demo configuration, following the Alexa+ MCP Toolkit authentication page: a built-in OAuth 2.0 authorization server with Tier 1 client_credentials (scope mcp:service, public tools) and Tier 2 authorization_code + PKCE S256 (scopes mcp:tools mcp:resources, caller-specific tools), no Dynamic Client Registration, no OIDC, 401 without WWW-Authenticate, single-use 60-second codes, ≤3600-second tokens, rotating refresh tokens, and user-level records bound to the token subject.
  • Simulated Alexa+ orchestrator: an agent loop that connects with the official MCP client, fetches a Tier-1 token, converts MCP tools to LLM tool schemas and includes an optional local Ollama model provider. The release video uses the bundled deterministic intent router so that the server workflow is repeatable; real MCP calls and OAuth are used throughout. When a tool answers insufficient_scope, it runs a real account-linking flow (consent page + PKCE) and retries with the Tier-2 token. Guards stop the model from inventing caller names or phone numbers and from claiming a booking it never made.
  • Quality: pytest suite covering every tool, refusal path, live HTTP, OAuth, scope enforcement, cross-user isolation, XSS escaping and the simulator journey; we deliberately broke the code to confirm the tests fail. Independent US AI code reviews were run before publishing, and its findings were fixed.

Challenges we ran into

  • Alexa+ developer-preview access and the alexa-ai CLI (private package registry plus an Amazon-granted role) are not available to a new developer, so we could not deploy to a device; we built the simulator the rules allow and kept the server Alexa-ready.
  • The Alexa+ docs describe two slightly different auth models (quickstart vs authentication page); we implemented the authentication page's two-tier model and logged the ambiguity in our friction log.
  • A 7B local model picks the right tool most of the time but sometimes invents details to complete a booking. Deterministic guards turned that into a follow-up question instead of a bad booking.

Accomplishments that we're proud of

  • The safety property ("the assistant can't confirm") is enforced by the server's shape, not by hoping the model behaves.
  • The repeatable demo exercises real MCP calls, service-to-user OAuth account linking, a pending hold, staff approval, and a confirmed-status readback. Automated checks cover failure paths as well as the happy path; these checks are not proof of production readiness.
  • A two-tier OAuth server, account linking and human approval all work end to end in the demo, without paid cloud inference.

What we learned

Voice agents need a "pending" state as a first-class concept. Holding, confirming and cancelling are different actions with different owners, and making that explicit in the tool surface makes the whole system easier to trust.

What's next for HoldDesk

Deploy as a real Alexa+ add-on once preview access is available (manifest template included); proactive notification when a hold is approved; MCP Apps widget for the approval queue on Echo Show; persistent storage and multi-office accounts.

Built With

  • fastmcp
  • model-context-protocol
  • oauth2
  • ollama
  • pkce
  • pytest
  • python
  • starlette
  • web-speech-api
Share this project:

Updates

Submission history