voice-web-mcp

Use a full duplex/speech to speech voice agent CLI to control websites via webMCP + naturally learning languages with accents/tones with Anki cards.

Comment

Inspiration

Customers sometimes need to access tools/websites, but they can’t do it because websites are hard. The websites have bad UX, the customers are older and non technical, or don't speak English.

What it does

Parts

I built two things:

  1. A basic webMCP controllable anki website for language learning.
  2. A (realtime duplex speech to speech) voice harness CLI that supports WebMCP.

Codex voice mode doesn't let you talk to websites, this thing does.

Examples

Anyways, here's me screwing around with a bunch of people's websites they built (besides the example video): https://www.youtube.com/playlist?list=PLPP6FwJZGwUA

  1. taboo game - https://www.youtube.com/watch?v=rqPqUa1mNcA
  2. rubiks cube https://www.youtube.com/watch?v=fIQoW5sJ65g
  3. more of my own anki - https://www.youtube.com/watch?v=a86MogHEFQA
  4. maze solver - https://www.youtube.com/watch?v=MVpu7fn3kqQ

More technical details

The website is straightforward, lets you import real Anki apkg files, and have it run on chrome.

The agent harness in a similar way to claude code but its streaming speech to speech.

  1. It can control your files, your website, chrome tabs/cast controls, etc.
  2. Voice wise it supports parsing like Barge in, accents, accoustic event cancellation, noise cancellation, high pass/gain controls, etc.

reference codes

  1. https://portpowered.github.io/anki-web-mcp/ is the website.
  2. https://github.com/portpowered/go-agent-harness is the CLI.

Note that you have to have spent like $50+ dollars on OpenAI APIs to get any real usage out of the voice CLI, otherwise you're going to get rate limited, so probably constrained to the webMCP website otherwise. But anyways, it should be able to access whatever website you want. The AI is kind of dumb though. It works on the various webMCP showcases.

Recordings:

  1. IRL recording: https://youtu.be/G_SsdL5Yqj8
  2. Screen recording: https://youtu.be/811ooWpI0Zw

How we built it

We built it largely using

1) Codex/Claude 2) An agent factory/orchestrator that wraps luna max/codex workers. I had a Claude agent manage the factory as appropriate. 3) OpenAI realtime API 4) Golang 5) NextJS based static site on github pages.

Challenges we ran into

  1. The UX of interacting with the MCP is highly fragile. You need to have to have a very strongly defined AGENTS.md and there is no current obvious way to declare WebMCP AGENTs.md/Instructions for each website.
  2. Tools that are available on different pages are different, and controlling them is hard for declaring the appropriate context on different pages.
  3. audio processing graphs are really actually quite complicated (barge in, accoustic event cancellation, umm/sigh, tool processing), this ended up being a significant time sink since current models aren't able to comprehend audio failures directly (except for like gemini). I don't really understand all the audio tools so was hard to build.
  4. Chrome is super flaky for casting and open/close with session controls.

Accomplishments that we're proud of

  1. its actually really cool to get the agent to just fix the site using your voice. I only recorded it for a second, but I was using it to learn japanese for a few hours since I usually run on my phone. It helps a lot with like coming with phrases and accent help. Helps a lot with korean and chinese as well, since my accent is bad, and I don't understand tones.
  2. ~90% of the code was implemented via luna max workers and fixed and dispatched by them. I mostly just left it to run on my laptop at about 50 agents or so.
  3. The digital signals processing is more work than I imagined and I was able to figure it out and understand. Understanding accoustic event cancellation, barge in, noise cancellation.
  4. Independent of the webMCP functions, the fact that there's now like a generic Anki page that customers can load their deck into is super useful, because installing an app is annoying.

What we learned

  1. webMCP is super useful actually, i'm probably going to tamper monkey a bunch of websites so that I can just buy stuff off doordash.
  2. Audio data signals processing, like how to make the voice robots not echo itself and re-trigger, how to set up the noise clearing, how to set up VAD, etc.
  3. Realtime models are much more useful than I anticipated, but they're still kind of flaky.

What's next for voice-web-mcp

  1. better multi tab handling if you have multiple browser tabs running and you want the agents to focus on a specific one or move between tabs generally.
  2. better multimodality input/output is needed for things that can watch/hear/read.
  3. better streaming is needed for things that happen over time
  4. better connections to payments and authorization is needed for delegated permissions
  5. better AGENTS.md/instruction declaration.

Built With

Share this project:

Updates

Submission history