Automatic Dealer

Tell Alexa+ what you want done on the phone. An AI agent calls in the other side's language, negotiates only within limits you confirmed, and asks you when a decision is yours.

Comment

Inspiration

I once booked a flight with an airline abroad and needed to change the date. The airline said changes go through its online process, but in practice, as with most changes, it still came down to a phone call. The agent and I didn't share a first language, and the call quickly turned into two people talking past each other. I built Automatic Dealer so that a simple change like that never has to end up lost in translation.

It isn't a rare problem. Some chores can only be done on the phone: cancelling a subscription that pushes a fee and a "final offer", returning an order to a shop abroad that only speaks Japanese, confirming a delivery window, or deciding whether an unknown caller is a scam. Voice assistants can tell you a company's number, but they can't make the call for you, and handing a phone call to an AI raises an obvious fear: what if it agrees to something you never wanted?

What it does

You tell the (simulated) Alexa+ host what you want, for example "Call Streamly Plus and cancel my subscription. I won't pay any fees." The assistant works out the scenario, the company and the likely language, and drafts a mandate: acceptable options, red lines, what it may share, and a payment limit. You confirm or edit it, and the agent makes the call.

  • Speaks the other side's language (Japanese, Chinese, English and more), with a live translated transcript.
  • Commitment gatekeeper. Every offer from the other side is extracted into a structured commitment and checked by a deterministic function against your mandate: allow, reject (red line) and counter, or escalate to you. The agent's own sentences are checked the same way before they are spoken.
  • Human in the loop. When something is outside the mandate, the agent asks the other side to hold and shows you a decision card (accept, decline, counter).
  • Relay mode. Press "Let me talk", type or say what you want, and the agent relays it in polite Japanese (or any supported language), adding nothing.
  • Inbound calls. Callbacks are matched to the waiting task and continued; callers covered by a standing rule are handled on their own; unknown callers get an incoming call card, and a scam is only ever taken as a message, never paid.
  • Memory. Outcomes, follow-up dates and per-company notes persist, so the assistant reminds you when a refund is due and calls again using what it learned.

The demo has five scenarios: cancel a subscription (English), a return with a manager callback (Japanese), a scam call, a delivery window call, and a proactive refund reminder. Every company on the other end is fictitious and played by our AI counterpart.

How we built it

  • MCP server built with the official MCP Python SDK: spec 2025-11-25 over Streamable HTTP at /mcp, bearer-token auth, 11 tools (draft_task, start_task, get_job_status, answer_decision, answer_inbound, set_mode, relay_message, list_history, cancel_job, get_settings, update_settings) and MCP Apps cards for the mandate and the live call status. In-call tools such as evaluate_commitment are deliberately not exposed, so no client can bypass the gatekeeper.
  • Simulated Alexa+ host: a Strands Agents agent with Claude that reaches the tools through a standard MCP client, exactly as an external host would.
  • Call pipeline: Amazon Transcribe streaming (with language identification) -> Claude for extraction, the gatekeeper and the reply -> a pre-speech commitment check -> Amazon Polly, sentence by sentence.
  • Two-step commitments: a model call extracts what was proposed (type, amount, option, confidence); then check(commitment, mandate) decides from those fields alone. Low confidence always escalates.
  • AI counterpart: each fictitious company is a persona with scripted beats (fees, "final" offers, callbacks, scams), phrased by Claude and voiced by Polly, so the demo moments always happen.
  • AWS: EC2 (Docker Compose) behind CloudFront, the web app on S3, DynamoDB single-table storage, S3 call recordings, Secrets Manager, CloudWatch, Systems Manager Run Command for one-command deploys, and AWS Budgets alerts.
  • Web app: Next.js with an Echo Show style layout, live call view, relay panel, history and settings.
  • Tests: 78 offline tests, including all five scenarios end to end over text and voice; every run asserts the gatekeeper allowed zero red-line commitments.

Trade-offs we made

  • A deterministic gatekeeper over model judgment. Safer and testable, but it escalates more often than a person would: an offer the extractor labels differently from your red line goes to you instead of being declined automatically. We chose an extra question over a wrong commitment.
  • A simulated Alexa+ host instead of a device. We could not load a custom MCP server into Alexa+ during the hackathon, so the host talks to the same MCP endpoint a real Alexa+ would.
  • An AI counterpart instead of real phone numbers. Reproducible demos and no calls to real businesses; a Twilio path is built but off.
  • Transcribe → Claude → Polly instead of speech-to-speech. Slower turns, but every sentence passes the pre-speech commitment check before it is spoken (and Nova Sonic was blocked with Bedrock).
  • Scripted persona beats phrased by Claude. The fee, the "final offer" and the callback always happen, at the cost of some realism.
  • EC2 with Docker Compose instead of Fargate. One small instance is cheaper and simpler for a demo; the Fargate stack is in deploy/ but not running.
  • Conservative defaults. A decision that times out is rejected, and if Claude is unavailable every step falls back to scripted behavior instead of failing.

Challenges we ran into

  • Amazon Bedrock was blocked for our whole account ("Error 002"), even for Amazon's own models, AWS Support escalated it to the Bedrock team, who said these are premium models that need a commercial agreement through AWS Sales and cannot be enabled self-service. Because every layer sits behind a small interface, we moved Claude to the Anthropic API and turned the Nova Sonic engine off without changing the rest of the system. The Bedrock and Nova Sonic paths remain implemented.
  • Keeping the model from agreeing to things. Early runs had the agent sounding like the company's own rep and a pre-speech check that misread a refusal as an acceptance. Splitting extraction from a deterministic gatekeeper, plus a labeled dataset of 120 lines in English, Japanese and Chinese, closed those holes.
  • MCP Python SDK 2.x reorganized its API, so mounting Streamable HTTP onto our existing FastAPI app took reading the package source.
  • A host deadlock: the Strands host connected to its own MCP endpoint synchronously inside the same event loop and waited on itself.

All of this is documented in our friction log in the repository.

Accomplishments that we're proud of

  • A full call, in Japanese, with a manager callback that the agent picks up and finishes on its own.
  • A safety model that doesn't depend on the language model behaving: the gatekeeper is a pure function with table-driven tests.
  • The whole system runs offline with stand-ins for every cloud service, so it is testable without AWS.

What we learned

  • Let the model talk, but never let it decide commitments.
  • Test with the live model early; offline fakes hid several real bugs.
  • Design for provider failure: a setting switch saved the project when Bedrock was blocked.

What's next for Automatic Dealer

  • Run it inside a real Alexa+ device through the MCP server.
  • Real phone numbers (a Twilio Media Streams path is implemented but off by default).
  • Nova Sonic speech-to-speech for English calls once Bedrock access is available.

Built With

Share this project:

Updates

Submission history