Hearsay: preflight checks for Alexa+ add-ons

Tests what your Alexa+ add-on says out loud: plays clean and misheard requests against your MCP server, grades every reply by Amazon's published rules, fails the build. Open source, unofficial.

Comment

Inspiration

An Alexa+ add-on is an MCP server. Tested by typing in a chat window, it looks fine. Spoken, it breaks in ways nobody sees until customers do: "fifteen dollars" is heard as "fifty", a reply of JSON is read aloud, a slow tool leaves seconds of silence, an order goes through on a yes nobody said. The Alexa+ MCP Toolkit and simulator are partner-gated, so hackathon builders can't run their add-on under Alexa+ itself. We wanted a crash test for voice add-ons that runs on every pull request.

What it does

Hearsay plays what customers say, clean and misheard, against your add-on over MCP, grades every reply by Amazon's published functional requirements for add-ons, and turns the pull request red where it breaks. No microphone, no model, no API keys, and the same result every time.

Every check answers one of four questions: Did it hear me right? Do I have to wait? Can I listen to this? Did I agree? Every finding names its source: Amazon's requirements (500 ms per tool call, replies a person can listen to, at most five options, confirmation with the key details before payment), the MCP spec, or Hearsay's own rules for budgets and consent.

Amazon's Local Inspector checks what your server declares. Hearsay tests what happens when a person talks to it, in CI, without a login.

How it works

  • Suites (YAML) say what a person says and what should happen. The runner connects over Streamable HTTP (MCP 2025-11-25), plays each case and perturbs what was heard with seeded mishearings (number confusions, homophones, split compounds). A literal stand-in planner carries the misheard words into the tool arguments, so CI needs no model.
  • The catalog turns Amazon's requirements into executable checks. A check counts as implemented only with a fixture built to fail it.
  • Consent tiers: a confirmation through MCP elicitation passes; a spoken question with a single-use token is graded weaker (warn); committing without asking is an error. A misheard amount must never commit.
  • Console: the simulated Alexa+ experience the FAQ describes. It is a web page whose engine is an MCP client (initialize, tools/list, tools/call over Streamable HTTP): talk to your add-on, hear the reply, answer its confirmation, and see the findings for each turn.
  • For coding agents: Hearsay is itself an MCP server (@hearsayhq/mcp) with two Agent Skills. A fresh Claude Code session with no shell turned our flawed Smart Home add-on green in about a minute for $0.35–0.44, changing server code only. It can't cheat: suites are locked, and holdout cases live outside the repo.
  • @hearsayhq/kit: the building blocks every finding's fix names: speak, refuse, looseEnum, looseInt, confirm, VerbalTokens, withMandate, serveMcp.
  • AWS: Amazon Bedrock (Converse, Amazon Nova 2 Lite) drives an optional llm planner and records replay cassettes. Amazon Polly and Amazon Transcribe record real mishearings over a phone-band channel. The results are committed and replayed, so CI never calls AWS.

Not just ours

We ran Hearsay on 111 other Alexa+ add-ons from this hackathon: public repos, pinned to a commit, built and run in containers with no network, no keys and no tests written for them. 106 had something to fix, and 19 had errors that fail a run. Of the 25 whose replies could be heard, 17 would read JSON, tool names or ids aloud, which Amazon's requirements rule out (docs/16; aggregates only, no names). If every add-on arrived with a green run, review could start where the checks end.

How we built it

TypeScript throughout (MCP TypeScript SDK 1.31, zod 4, Hono, Vite + React), spec first: a docs pack with requirement ids, a decision log and a risk register. There are three reference add-ons:

  • Kitchen: what green looks like.
  • Smart Home: flawed by default; applying the kit makes it green.
  • Household Orders: a spending permission behind a voice assistant.

Challenges

  • Amazon doesn't document whether Alexa+ supports elicitation, and its example handshake declares none, so consent had to be graded instead of assumed.
  • The MCP SDK answers invalid input with "MCP error -32602" before the handler runs, and a person would hear that. The kit advertises enums and bounds on a loose schema and refuses in words.
  • Hearsay's own skill found two such bugs in our reference Kitchen server on its first draft.

What we learned

We measured whether Hearsay helps a coding agent. Without it, 11 of 27 agent runs ended with defects the suite would have caught; with it, 0 of 36. But a suite alone made agents stop at green: on unseen cases they did worse than agents given a precise prompt (68–71 % vs 83 %). So Hearsay now reports what your suite doesn't cover, and its skill reviews beyond green.

Measured again, Hearsay was:

  • level with the precise prompt on unseen cases (84 % vs 83 %);
  • ahead on rules the prompt never mentioned (37/45 vs 28/45);
  • leaving zero defects behind (docs/15).

Real recordings taught us something too: in 240 transcriptions over a noisy phone line, Transcribe never confused a number. It dropped verbs and swapped nouns instead.

What's next

  • MCP 2026-07-28 input_required for the kit's confirm()
  • persona-driven multi-turn goals
  • a composite GitHub Action

Details are in docs/ROADMAP.md.

Try it

No keys, no account; about a minute from a fresh clone.

git clone https://github.com/hearsayhq/hearsay.git && cd hearsay && npm ci
HEARSAY_FIXED=0 npm run hearsay -- run suites/household-orders.yaml   # flawed add-on: 20 errors, exit 1
npm run hearsay -- run suites/household-orders.yaml                   # fixed add-on: 0 errors, exit 0

In your own project: npx @hearsayhq/cli run suites/*.yaml. The README lists the commands behind every scene of the video.

Built With

  • agent-skills
  • alexa-plus
  • amazon-bedrock
  • amazon-nova
  • amazon-polly
  • amazon-transcribe
  • claude-code
  • docker
  • github-actions
  • hono
  • mcp-typescript-sdk
  • model-context-protocol
  • node.js
  • npm
  • react
  • typescript
  • vite
  • vitest
  • zod
Share this project:

Updates

Submission history