Voice Turn: Listening Care With Spoken Alexa Command
Architecture Slide
MCP Elicitation Confirm Card
Rule-Checked Replan Proposal With Reasons
Hash-chained Audit Log
Schedule Applied, v3

Inspiration

Universities run expensive instruments, such as confocal microscopes, as shared, bookable core facilities, and they bill the time by the hour. For example, Duke's Light Microscopy Core charges $38.50 per hour for laser-scanning confocals and charges an unused reservation for the whole booking (Duke fees). Penn's CDB Microscopy Core lists $52 per hour for unassisted peak-time confocal use (Penn rates).

When one of those instruments fails in the middle of the day, every booking that overlaps the outage breaks at once. Someone, usually the facility manager, has to rebook each person by hand, checking opening hours, who is trained on which instrument, each person's other commitments, deadlines and cleaning gaps between sessions. That is a multi-step, rule-heavy task. A voice assistant can help, but only if it cannot quietly make the wrong change.

The idea and concept came from our teammate Chris Yu.

What it does

You tell the assistant what broke, and it brings back a plan you can check. Nothing changes until a human confirms.

Voice turn (simulated Alexa+). Press push-to-talk and say "Alexa, Confocal A is down until 4pm, please sort out the bookings." The browser's Web Speech API transcribes it (the wake word "Alexa" is required), and the reply is shown and spoken aloud.
Bounded Amazon Bedrock tool loop. When Bedrock is on, Amazon Nova (Converse API) turns the sentence into MCP tool calls: report the outage, propose a replan, ask to apply. The server keeps it on a short leash: an allowlist of 5 tools (destructive and debug tools are refused), at most 4 Converse rounds and 4 tool executions per conversation, per-token and per-IP budgets, and a 20 s timeout. Any malformed call, unknown tool or schema mismatch stops the loop with nothing executed.
A deterministic engine decides, not the model. The replan engine searches 15-minute slots, earliest deadline first, and checks opening hours, per-person instrument permissions, personal calendars, hard deadlines and a 15-minute cleaning buffer. Every move carries a reason; when a booking can't be saved, it explains what would make it fit.
A human Confirm is mandatory. Applying a plan sends a real MCP elicitation/create request that only a human click answers. Cancel changes nothing, and a client without elicitation support gets not_confirmed with zero side effects.
Accountability. Every tool call, model proposal and Bedrock attempt goes into a SHA-256 hash-chained audit log. Identity comes only from the auth header, never from tool arguments. Researchers see their own bookings while others show as "Occupied". The schedule is versioned (v1 → v2 → v3), stale plans are rejected and a second apply returns already_applied.
Labelled fallback. If Bedrock is off or errors, the same turn runs through a small pattern parser, and the UI says so, e.g. "Fallback: pattern parser (Bedrock off)". The demo video runs in this mode.

How we built it

MCP server (Python): the official mcp SDK (FastMCP), spec 2025-11-25, over Streamable HTTP with SSE (needed because elicitation is a server-to-client request). 8 tools, with a frozen contract and end-to-end fixtures replayed byte for byte in tests. Engine and state: a pure, deterministic Python replan engine on SQLite, with a monotonic schedule version, idempotent plan IDs and the hash-chained audit trail. Web MCP client (the simulated Alexa+ experience): HTML/JS using the official TypeScript MCP SDK, talking through a FastAPI same-origin proxy that relays SSE live so the Confirm card appears instantly. A protocol console shows the raw MCP traffic. Voice: browser SpeechRecognition for push-to-talk and speechSynthesis for replies; a ?utterance= hook drives the same code path for tests and recordings. Amazon Bedrock: boto3 bedrock-runtime converse with a toolConfig built from the server-side allowlist (default amazon.nova-lite-v1:0), with strict timeouts and an audit row per attempt (model, region, latency, request ID; never the token). Hardening: Origin/Host allowlists against DNS rebinding, per-IP rate limits that return proper JSON 429s, and a /metrics endpoint. Testing: pytest (engine, contracts, a real server with a real MCP client, the proxy, the agent loop against a stubbed Bedrock client), Playwright browser walkthroughs, and GitHub Actions on Python 3.11 and 3.13. Team and AI assistance. Chris Yu came up with the idea and product concept. Hao En Fu did the improvement and implementation: the server, client, tests and Bedrock integration, while directing and reviewing AI coding agents, which assisted development (many commits are authored by AI agents). Independent AI-run verification passes checked every change before merging.

Challenges we ran into

Elicitation silently disappeared. With FastMCP's JSON-response mode, the Confirm request never reached the client and the tool just returned not_confirmed. Server-to-client requests need SSE, so we switched transports. The Bedrock account hold (still open). On our new AWS account, plain Converse calls worked on 21 Oct (e.g. Nova Lite, us-east-1, request ID dd109776-e897-4f37-be97-6e6356e78357; Nova Pro, eu-west-2, request ID f7215dd1-c7c2-404a-95ab-071b46d5a40e). Shortly after, every model in every region began returning Operation not allowed, with authorizationStatus=NOT_AUTHORIZED and "Your account is currently being verified". We opened an AWS Support case and are still waiting. Failing closed under real errors. Because live Bedrock broke mid-build, we made every Bedrock error (throttling, validation, access denied, network, timeout) fall back visibly, labelled in the UI and audited with its request ID. Our own tests lied to us once. An early browser test could never fail because of a helper's argument order. Fixing it exposed four real product bugs. A later adversarial review found the model could reach a destructive tool without a Confirm; the server-side allowlist closed that. Our rate limiter blocked our own agent. A single voice turn makes many quick MCP requests, which tripped the per-IP limiter and returned an invalid 429 body. We gave the agent its own bounded budget and fixed the 429 JSON.

Accomplishments that we're proud of

Human-in-the-loop is enforced by the protocol, not by a prompt. Neither the model nor the client can confirm on a person's behalf. A stubbed model that forges confirm: true, calls apply directly, asks for a destructive tool or sends malformed calls is refused, and each refusal is audited. Measured, not claimed: 111 tests passed (1 skipped) with the browser suite; CI green on Python 3.11 and 3.13; gitleaks clean. Honest about the AI layer. The UI always shows which path ran, and the README has a "What is real, and what is simulated" section.

What we learned

Elicitation is the right primitive for any agent that writes, but transport settings can silently disable it, so servers need a safe path for clients that don't support it. Let the model propose and the engine decide. A small, server-enforced tool allowlist and step caps are worth more than prompt instructions. A new cloud account can be blocked at the account level even when permissions look right. Plan for it with a visible fallback and request-ID audit trail. Test your tests: checking "does this fail if I inject the bug?" found real defects.

What's next for Shared Equipment Rescue

Live Bedrock, re-verified. Once AWS lifts the hold, run the live tool loop and golden path repeatedly and publish the request IDs. Amazon Nova Sonic speech-to-speech in place of browser Web Speech, for a real bidirectional voice turn. A real Alexa+ device, packaging the server for Alexa+ once that path opens to developers. Real booking systems (e.g. a core facility's scheduler export) and multi-day replans. Not built yet, and not piloted with a real facility.