Alexa- Voice Recognition (Voice Gate)

Inspiration

Voice assistants trust anyone who can be heard. A guest, a child, a prankster or even a TV ad can say "unlock the front door" and the assistant will try to do it. That was tolerable when assistants only played music, but modern assistants like Alexa+ call real tools through protocols such as the Model Context Protocol (MCP): smart home, reminders, schedules, shopping and more.

We asked a simple question: what if the assistant checked who is speaking before it was allowed to use any tool?

What it does

Voice Gate for Alexa+ puts a biometric checkpoint in front of an MCP-powered assistant.

Verify the voice. Audio is turned into a 192-dimensional speaker embedding and compared with each enrolled user's voiceprint.
Mint a short-lived token. Only if the match passes does the system issue a 60-second signed token. Unknown voices are denied and never reach the MCP server.
Protect the tools. The MCP server rejects any request without a valid token (HTTP 401), and tools learn who the caller is from the server context, never from arguments the client or LLM could fake.
Apply tiered access control. What a verified (or unverified) speaker can actually do depends on how sensitive the action is:

Tier	Examples	Requirement
0	Emergency ("call 911")	Always allowed
1	Music, weather, general info	Guests allowed
2	Smart home, TV, timers	Enrolled users
3	Locks, intercom, shopping, routines	Higher confidence
4	Payments, device settings, guest permissions	Very high confidence + voice PIN

Personal data (calendar, reminders, notes) is strictly per-user. Cross-user sharing is off by default, must be granted explicitly by the owner, and every cross-user attempt is audited.

How we built it

Speaker recognition: SpeechBrain's ECAPA-TDNN produces embeddings. Each enrolled user is stored as a centroid, and a request is scored with cosine similarity:

$$ \text{score}(\mathbf{x}, \mathbf{c}) = \frac{\mathbf{x} \cdot \mathbf{c}}{\lVert \mathbf{x} \rVert \, \lVert \mathbf{c} \rVert} $$

Access is granted when $\text{score} \ge \tau$, where $\tau$ depends on the sensitivity tier ($0.40$, $0.65$ or $0.80$).

Tokens: HMAC-SHA256 signed bearer tokens with a 60-second TTL.
Servers: a FastAPI app (web UI and /api/ask) and a separate self-hosted MCP server over Streamable HTTP, both launched with one command (python run.py).
Identity isolation: middleware validates the token and stores the caller in a ContextVar. All 17 tools read identity from there.
Access control layer: an intent classifier maps each utterance to a category and tier, and a policy enforcement gate decides to allow, deny or challenge (PIN or confirmation). Multi-speaker or ambiguous audio is automatically demoted to guest.
Tooling: a text-based simulator for testing policies without audio, an automated test suite (26 tests, fully offline), a Docker setup, and a threshold evaluation script.

Challenges we ran into

Alexa doesn't expose raw audio. Real Alexa hardware does speech capture internally, so third-party skills can't run their own speaker verification. We built a simulated Alexa+ web experience that gives us raw audio while keeping the MCP side spec-compliant.
Choosing a threshold. A low threshold lets impostors in, and a high one locks out real users. We built an evaluation script that measures the False Acceptance Rate and False Rejection Rate:

$$ \text{FAR}(\tau) = \frac{\text{impostor attempts accepted}}{\text{impostor attempts}}, \qquad \text{FRR}(\tau) = \frac{\text{genuine attempts rejected}}{\text{genuine attempts}} $$

It reports the Equal Error Rate point, where $\text{FAR} = \text{FRR}$, to guide the default.

Never trusting the client. Letting an LLM or client pass "who I am" to a tool is an easy way to leak data, so identity is derived only from the verified token.
Refusals that don't leak. A denial must not reveal whether another user's data exists, so guest refusals are polite and generic.
Replay attacks. A recording of a real voice can fool plain similarity matching, so we added an optional liveness mode that asks the speaker to repeat a random 3-word challenge phrase.

Accomplishments That We're Proud Of

True Zero-Trust Gating: Ambient voices that fail verification ((S_C < \tau = 0.40)) are blocked at the gate, resulting in 0 unauthenticated MCP tool calls and zero sensitive data leaks.
Cryptographic Token Lifecycle: Designed a lightweight token-minting pipeline using HMAC-SHA256 with an auto-expiring TTL of (\Delta t \leq 60\,\text{s}), tightly bounding the replay attack window.
192-D Neural Voice Biometrics: Built on-device acoustic verification using ECAPA-TDNN embeddings ((\mathbf{e} \in \mathbb{R}^{192})) with centroid stabilization across (N \geq 3) clips, achieving sharp speaker separation.
Spoof-Proof Identity Isolation: Eliminated prompt-injection spoofing by binding caller identity strictly to server-side ContextVar state rather than trusting LLM tool arguments.
26/26 Offline Passing Tests: Validated the entire system—including token tampering, multi-tier permissions, and data partitioning—with 100% of offline tests passing, requiring zero cloud API keys or GPUs.
Privacy-by-Design: Raw audio is processed entirely in volatile RAM and discarded; only irreversible mathematical embeddings are stored, alongside a one-click revocation API (DELETE /api/users/{name}).

What we learned

Authentication and authorization are different problems: knowing who is speaking is not the same as deciding what they may do.
Default-deny sharing and a single authorize() choke point make security much easier to reason about and test.
Biometrics are probabilistic, so a threshold per sensitivity tier, plus a second factor for the riskiest actions, is more honest than one global cutoff.
Privacy has to be designed in: explicit consent, no raw audio stored (only embeddings), and one-click voiceprint deletion.