Inspiration
Ask Alexa about a product recall or a drug-label warning and you want the answer from the source, word for word. LLM assistants cite confidently, and the citation is often not on the page. We measured it: a plain chat model answering live recall/drug-label questions produced 5 / 5 citations whose quote was not on the cited page.
What it does
Receipts for Alexa+ is a self-hosted MCP server (spec 2025-11-25, Streamable HTTP at /mcp) exposing one tool, verify_citation(url, quote, claim). The server fetches the cited page itself (no redirects followed), converts HTML/JSON to text, and checks the quote verbatim. It returns verified, unverified (with the closest passage on the page and its word overlap) or unreachable, plus the exact sentence Alexa should speak.
The rule: an unverified claim is never spoken. Alexa says "I couldn't confirm that from the source, so I won't say it."
Results (all reproducible with one command)
120-case eval built from live government sources (CPSC recalls, DailyMed drug labels, ENERGY STAR): 60 genuine quotes, 20 misquotes (one fact flipped), 15 misattributed (real quote, wrong page), 15 invented, 10 dead URLs.
| arm | fake citations spoken | genuine spoken |
|---|---|---|
| ungated (control) | 60 / 60 | 60 / 60 |
| Receipts gate (over MCP) | 0 / 60 | 60 / 60 (0 false rejects) |
Live agent arm (Strands Agents + Receipts MCP): plain chat spoke 5/5 unsupported citations; through Receipts 0/4, with 1 honest abstention.
How we built it
- Go,
modelcontextprotocol/go-sdkv1.6.0, Streamable HTTP transport, protocol 2025-11-25. quotecheck/: standalone verifier library (HTML/JSON → text, normalization, verbatim span match, near-miss window).cmd/demo: a real MCP client that runs the whole eval over Streamable HTTP against a fixture cache —make judge-demo, ~3 s, no keys, no network.agent/agent_eval.py: Strands Agents loop that calls the MCP tool, for the live-agent arm.
Challenges
- A government recall API returns HTTP 200 with
[]for a recall that doesn't exist — "200" is not "source exists". We treat empty payloads as unreachable. - A 74%-overlap near-miss with a fabricated attribution ("according to the manufacturer's latest safety bulletin") would pass any fuzzy threshold. Verbatim matching is the point: close is not true.
- Bedrock model access on a fresh account failed with an opaque
Error 002(documented in our friction log).
Honest limits
Genuine quotes are verbatim by construction, so 0 false rejects is a floor, not a field result. A model that paraphrases will be rejected — by design, Alexa abstains rather than speaking an unverifiable paraphrase. The live-agent arm is small (n=5 completed); provider errors are reported, not hidden.
What's next
Agent Skill packaging for Alexa+, an MCP Apps evidence card, and a larger live-agent run.
Product feedback / friction log
See FRICTION-LOG.md in the repo.
Log in or sign up for Devpost to join the conversation.