ShopVoice

Inspiration

Vietnam has more than a million family-run "tạp hóa" corner shops. The owner is cashier, stock clerk and buyer at once, usually with both hands busy behind the counter. They know roughly what is running low, but reorders happen from memory, on paper, often too late, and they always forget the one rule they care about ("eggs come from the farm supplier, not the dairy").

Before the hackathon, our project GroceryClaw automated one painful thing for them: turning supplier invoices (photo, XML, Excel) into the KiotViet point-of-sale system. During the hackathon we gave them what they actually ask for all day: a shop they can run by voice, with an agent that watches it for them.

What it does

The owner talks, the agent acts:

• Morning check. "Good morning, how's the shop?" The agent chains three steps in one answer: yesterday's sales against the same weekday, what is running low, and a reorder plan grouped by supplier with the total. Then it asks: "Draft Dawn Bakery & Farm first?"
• Supplier by supplier. Each "yes" drafts one supplier's order; a second "yes, confirm" places it, and the agent offers the next supplier from the plan.
• Shop memory across days. "Always order eggs from Dawn Bakery & Farm." The agent repeats the rule back and applies it from then on, saying so: "As you asked: eggs from Dawn Bakery & Farm."
  • It learns only from explicit statements. "What do you remember?" lists the rules, and "Forget that" deletes one.
  • Supported rules: preferred supplier, an order cap ("never more than 40 eggs"), items to skip, and the usual order day (mentioned in the morning briefing).
• Everyday questions:
  • "How were sales today compared to last Friday?"
  • "Did the Sunrise Beverages invoice arrive?"
  • "How many eggs do we have?" For an ambiguous product it asks "which one?" instead of guessing.
• Vietnamese. "Mặt hàng nào sắp hết?" gets a Vietnamese answer from the same tools with the same safety rules. "Có, xác nhận" confirms; "Đơn này bao nhiêu tiền?" is a question, so it never confirms.
• Safety. Nothing is ever ordered without an explicit spoken yes to a draft the owner has already heard. The model never sees the confirmation token.
• MCP Apps view. In hosts that render MCP Apps, suggest_reorder comes with an interactive plan (ui://shopvoice/reorder-plan): one table per supplier with editable quantities, and "Draft" then "Confirm" buttons. Hosts without MCP Apps keep the spoken answer, which is complete on its own. We checked the view in a sandboxed-iframe host harness (evidence/mcp-app-reorder-plan.png); the voice simulator does not render it.

Before → after

How we built it

• MCP server (apps/mcp-server):
  • TypeScript on Node 22 with raw node:http and the official @modelcontextprotocol/sdk.
  • Streamable HTTP; negotiates protocol 2025-11-25.
  • Tools:
  • reads: get_low_stock, get_stock_level, get_sales_summary, get_top_movers, get_invoice_status, suggest_reorder, get_daily_briefing, list_shop_rules;
  • writes: create_reorder_draft → confirm_reorder (5-minute hashed token), remember_shop_rule, forget_shop_rule.
  • Also a morning_briefing prompt and a shop://profile resource.
  • Every tool returns one or two spoken sentences (35 words at most) plus structuredContent matching its declared outputSchema, with read-only, destructive and idempotent annotations.
• Data:
  • Postgres 16 with row-level security, one tenant per shop. Every call runs in a tenant-scoped transaction and is written to an audit log.
  • The hosted server logs in as a NOSUPERUSER NOBYPASSRLS role and refuses to start as anything more privileged.
• Simulated Alexa+ experience (apps/alexa-sim):
  • A push-to-talk web page (Web Speech API, en-US and vi-VN) whose server is a real MCP client: initialize, tools/list and tools/call over Streamable HTTP.
  • Claude Sonnet 5.5 (Anthropic API, low effort) plans the tool calls. A deterministic rules brain takes over within the same turn if Claude errors, times out, refuses or the daily budget is spent, so judges never see an error page.
  • A side panel shows every MCP call with its arguments and latency.
• Host-side safety:
  • The host holds the confirmation token and injects it only when the owner's own words in this turn are a yes to a draft created in an earlier turn.
  • The host also fills in who is speaking, and the model never sees that field, so one person's rules cannot leak into another's.
• MCP Apps: suggest_reorder declares _meta.ui.resourceUri = ui://shopvoice/reorder-plan. The server serves a self-contained text/html;profile=mcp-app view that speaks the MCP Apps postMessage JSON-RPC: ui/initialize, tool-result, size-changed, and proxied tools/call. We did not add the ext-apps helper package, because its current 2.x line requires the v2 SDK and this server runs SDK 1.x.
• Hosting: one Render Blueprint creates:
  • a private Postgres 16 service with a disk;
  • the MCP web service, with a pre-deploy migration and role setup;
  • the simulator web service on the private network;
  • a nightly cron that re-anchors the demo shop to "today", so "last Friday" always has data during judging.
• Tests: node:test throughout: 271 unit tests plus 69 integration tests against real Postgres and Redis, all run by v2-ci on every PR. Covered:
  • protocol negotiation, auth, scopes, Origin checks, rate limits;
  • tool contracts, the 35-word rule, RLS isolation and token expiry;
  • the host's confirmation guard in English and Vietnamese;
  • memory scoping, the morning plan, and the deploy scripts against real Postgres.
• Claude evals: 31 conversations, 53 utterances, against the hosted demo. Hard gates: no confirmation without an explicit yes, no rule saved unless the owner stated one, no number spoken that a tool did not return.

Challenges we ran into

• Our AWS account was locked, and the appeal failed. Amazon then ran out of credit codes. We did not open a second account. Instead, we redesigned the live path around what we could run honestly:
  • Claude via the Anthropic API instead of Bedrock;
  • browser speech instead of Polly;
  • Render instead of EC2 and CloudFront.

The Bedrock, Polly and CDK code stays in the repo, labelled "implemented, not deployed: AWS account unavailable".

• Managed Postgres has no superuser. Our RLS bootstrap needs BYPASSRLS for a definer-function owner, so Postgres runs as a Render private service. A pre-deploy step creates an RLS-bound runtime role.
• Voice answers have to be short and true. Every tool builds candidate sentences and speaks the most detailed one that fits 35 words. The prompt allows only numbers that came from a tool, and the evals check every spoken number.
• Memory in a shared demo. One judge saying "skip milk" must not break the next judge's demo. Rules are therefore keyed by the speaker's voice profile, which the host sets and the model never sees.
• A "yes" in two languages. "Có đơn nào chưa?" contains "có" (yes) but is a question. The host's confirmation guard learned Vietnamese question particles.

Accomplishments that we're proud of

• An agentic workflow, not a single-turn Q&A bot: briefing → plan → draft → confirm, supplier by supplier, with memory that persists across days.
• Real MCP conformance: the official MCP Inspector connects to the hosted server, shows protocol 2025-11-25, and runs every read tool with the public read-only token (writes are refused with 403 insufficient_scope).
• Hosted latency from outside: MCP read tools p50 256 ms, p95 286 ms (n=70); a full voice turn with Claude p50 3.2 s, p95 4.2 s (n=53).
• Claude evals on the hosted demo: safety 53/53, spoken numbers 53/53, tool choice 48/48, e2e 5/5, no fallbacks, about $0.27 for all 53 utterances. A first run on an 8 s budget timed out twice (safety still 53/53); we raised the budget to 11 s and kept both runs in the evidence.

What we learned

MCP is the right seam for voice. The same server serves a voice simulator, the MCP Inspector and any assistant without change. For voice, the tool's output matters as much as the model: spoken text plus structured data plus annotations is the contract that keeps answers short and actions safe. Safety belongs in the host, not the prompt: the model can propose, but only the owner's own words can confirm.

What's next

• Live KiotViet inventory and sales through the existing worker adapter, and sending confirmed drafts to KiotViet POST /purchaseorders.
• Rendering the MCP Apps reorder view inside the simulator too, next to the voice answer.
• A real Alexa+ integration once partner access opens.
• Vietnamese spoken directly by the tools (today Claude translates them).
• Moving the hosting back to AWS when an account is available; the CDK stack is already written.

Built With

• anthropic-api
• claude
• docker
• ffmpeg
• kiotviet-api
• mcp-inspector
• mcp-typescript-sdk
• model-context-protocol
• node.js
• piper
• playwright
• postgresql
• render
• row-level-security
• typescript
• web-speech-api
• zod