EchoProbe

An MCP server that audits voice-assistant tools for prompt injection, privacy leaks and speech-unfriendly answers

Comment

Inspiration

Voice assistants like Alexa+ are starting to call outside tools through the Model Context Protocol (MCP). That is powerful, but it opens a new attack surface: a tool can hide instructions in its output or even in its own description and trick the assistant into leaking private data. On top of that, answers written for screens (markdown, links, long text) sound awkward when spoken aloud. We found no simple way for a developer to test a voice tool for either problem, so we built one.

What it does

EchoProbe is a self-hosted MCP server (Streamable HTTP, protocol 2025-11-25) plus a one-command audit toolkit for voice-assistant tools.

  • Prompt-injection firewall: a rules layer, plus an optional LLM judge (Gemini) meant to catch paraphrased and non-English attacks that rules miss
  • Tool-poisoning audit: scans tool names and descriptions for hidden instructions
  • Confirmation gate: risky actions such as sending a message are held until the user confirms
  • Voice readiness: a voice score, a rule-based spoken rewrite, a latency check, personal-data detection (emails, phone numbers, cards, PINs) and a link allow-list
  • Simulated users: a hurried user, a typo-prone user and an attacker test any MCP server with and without protection
  • Local web app and report: a simple UI with a try-it scanner, a full audit, grades, an audit log and a JSON export

How we built it

Python, the MCP Python SDK (1.x) and a small standard-library web server for the UI. We wrote a deliberately flawed demo server (poisoned tools, a screen-style answer, a slow tool, one tool leaking personal data) so every feature has something real to detect. A hand-written benchmark covers 20 attacks (including Bangla, Hindi, Spanish and French) and 20 harmless outputs. The optional LLM judge uses the Gemini API, with timeouts, retries and a circuit breaker.

Challenges we ran into

  • SDK version break: the MCP SDK 2.x renamed the APIs we started with, so a fresh install failed. We pinned mcp<2.
  • API quota: our free-tier Gemini quota ran out during testing, so the LLM judge could not be measured. We say so openly instead of guessing a number.
  • Hangs on failures: slow API retries made our own test suite hang, so we added short timeouts and a circuit breaker that falls back to rules only.
  • No real Alexa+ device: the agent in our demos is simulated.

Accomplishments that we're proud of

  • A working MCP server and an end-to-end audit that runs with one command, including a clear web app
  • Honest reporting: on our small benchmark the rules alone caught 6 of 20 attacks (30%) and wrongly blocked 2 of 20 harmless outputs
  • A multilingual attack set, and a tool-poisoning check that reads tool descriptions, not just outputs
  • A design that keeps working when the LLM is unavailable

What we learned

Keyword rules are not enough for prompt injection: they miss paraphrased and non-English attacks, and they can block harmless advice such as "never share your password". We also learned to design for failure (quotas, rate limits, version changes) and to report limits plainly.

What's next for EchoProbe

  • Measure the LLM judge properly once we have more API quota, and grow the benchmark and its languages
  • Finish the experimental drop-in proxy, so any MCP server can be protected by changing only its URL
  • Test against a real Alexa+ device and real third-party MCP servers
  • Run the audit automatically in CI for every tool update

Built With

Share this project:

Updates

Submission history