Inspiration
Most AI integrations treat the browser as a remote control: click this button, fill that form, or navigate to that page. We wanted to explore a more WebMCP-native question:
What important context exists inside the browser that an agent cannot access just as effectively from a terminal or traditional MCP server?
Live audio is one answer.
Traditional browser automation surfaces such as the Chrome DevTools Protocol are excellent at exposing the DOM, network activity, screenshots, and input events, but live collaborative audio is not naturally represented as page state or a sequence of browser actions. Hear My Site turns that missing real-time modality into incremental semantic context that agents such as Codex can consume while the event is happening.
While watching a tutorial, webinar, or online meeting, the human continuously receives information through the browser, but their agent hears nothing. Sharing that context usually requires recording the session, uploading audio, waiting for a transcript, or repeatedly copying captions into a chat.
We wanted the browser to expose this transient, browser-native context directly, allowing the human and agent to attend to the same live event at the same time.
The project began as Hear My Browser, an extension that works on websites we do not control. We then expanded the idea into Hear My Site: a headless package that lets websites provide the same capability natively, without requiring every visitor to install an extension.
The result is not simply “MCP inside a browser.” The browser is part of the problem because it owns the live media streams, user permissions, page lifecycle, and shared experience that the agent needs to understand.
What it does
Hear My Site gives AI agents ears for browser applications.
A website supplies its existing audio streams—such as local microphone audio and remote WebRTC tracks—and Hear My Site mixes and transcribes them using the provider configured for that experience. It currently ships with adapters for Gemini 3.5 Transcribe Live and OpenAI GPT-Live-Transcribe while keeping the agent-facing interface independent of either provider.
Across both integrations, Hear My Site exposes exactly three WebMCP roles: start listening, retrieve incremental context, and stop listening. Hear My Browser uses the default names:
start_browser_transcriptionget_browser_transcriptstop_browser_transcription
Adopting sites can rename all three roles to match their product. Our meeting demo exposes them as join_meeting, follow_meeting, and leave_meeting.
An agent can start listening, retrieve only the transcript segments it has not seen before, and stop when the task is complete.
The incremental cursor-based interface is important for long sessions. Instead of repeatedly sending an entire meeting or lesson transcript into the agent's context window, the agent retains the latest cursor and requests only new segments.
Our meeting demo shows the complete experience. Up to five people can join a lightweight WebRTC room, and each browser owns its own local transcription session. WebMCP joins through an explicit tool call, the room shows its presence and bounded live captions, and visitors never need to enter a transcription credential.
This enables interactions such as:
- “Keep listening to this lesson and wait until I ask a question.”
- “What explanation did the speaker just give for event loops?”
- “Summarize the decisions made since the last time you checked.”
- “Which questions from this meeting have not been answered yet?”
Hear My Browser provides the same tools for YouTube, webinars, and meeting websites that have not integrated Hear My Site themselves.
How we built it
We organized the project as a pnpm monorepo with three parts:
- A browser-only, framework-agnostic TypeScript package containing the shared transcription core.
- A WXT Manifest V3 extension that adapts Chrome tab capture to that core.
- A React and Vite meeting demo using WebRTC, a Cloudflare Worker, and Durable Objects for signaling and short-lived credential provisioning.
The shared core separates browser audio capture, transcription transport, transcript storage, and WebMCP registration.
Audio sources are dynamically connected through a Web Audio graph. An AudioWorklet converts the mixed signal into the format required by the selected transcription provider and emits small chunks suitable for low-latency streaming.
Provider adapters handle Gemini 3.5 Transcribe Live and OpenAI GPT-Live-Transcribe. They normalize different authentication methods, session setup messages, audio formats, interim results, finalized utterances, errors, and connection lifecycles into one shared transcription state machine.
Manual UI controls and WebMCP tool calls operate the same session and transcript store. The website can change providers without changing the three tools or the cursor format used by agents.
Tools are registered with the current imperative WebMCP API through document.modelContext.registerTool(...).
The extension keeps browser-specific responsibilities—tab capture, popup controls, Chrome storage, offscreen document lifecycle, and tab ownership—as adapters around the shared package.
The meeting demo uses full-mesh WebRTC for up to five participants. Its Cloudflare Worker and Durable Object coordinate room membership, SDP, and ICE signaling, while the Worker also provisions constrained, single-use Gemini ephemeral tokens from a server-side secret. They never receive media or transcript text, and the long-lived key is never returned to the browser. Meeting media remains peer-to-peer, while transcription audio travels directly from each browser to Gemini.
Challenges we ran into
The first challenge was browser lifecycle management. Chrome requires a real user gesture before tab capture can begin, while an agent may invoke a WebMCP tool later. We designed the popup, background worker, offscreen audio document, and tool errors so this security boundary remains explicit instead of hiding it behind an unreliable fallback.
Supporting multiple live transcription services required more than changing an endpoint. Providers differ in audio formats, authentication, session initialization, WebSocket events, connection limits, and how they represent interim and finalized text.
We created a provider boundary that normalizes those differences without leaking provider-specific behavior into the WebMCP tools. This gives websites and agents one stable contract even as transcription services evolve.
Long-running transcription introduced another challenge. Provider connections may expire or need rotation, but a lesson or meeting can continue much longer. The transport can reconnect while preserving the higher-level session, accumulated transcript, and agent cursor.
Audio ownership was also subtle. Extension-captured tracks should stop when transcription ends, but WebRTC tracks borrowed from a meeting must continue carrying the call. The shared package therefore tracks ownership explicitly and cleans up only the resources it owns.
Finally, WebMCP is still experimental. We deliberately target the current document.modelContext API and return a clear unsupported-browser error rather than shipping a deprecated alias, polyfill, or unverified fallback path.
Accomplishments that we're proud of
We are proud that Hear My Site is more than a browser automation demo. It brings a live, collaborative audio modality into agents such as Codex and gives them semantic access to transient browser-native context that would otherwise require a separate recording, upload, and backend transcription pipeline.
We are also proud of keeping the agent interface intentionally small. Three tools express the complete lifecycle, while the cursor protocol makes continuous monitoring practical for real agent context windows.
Hear My Site is not tied to one AI vendor. Gemini and OpenAI can both power the transcription layer, while websites and agents depend on one stable WebMCP contract. The provider is an implementation choice; the human-agent collaboration model remains the same.
The extension, headless package, website UI, and WebMCP tools all operate the same live transcription state. In the product runtime, there is no second agent-only implementation that can drift from the user experience.
Privacy is reflected in the architecture rather than only described in a policy. The meeting Worker never receives media or transcript text and never returns its long-lived Gemini key to the browser; the headless package does not persist credentials; and spoken content is identified as untrusted data rather than agent instruction.
What we learned
We learned that WebMCP is most compelling when the browser itself is part of the problem—not merely a convenient place to host an MCP server.
The live audio, WebRTC tracks, user gesture, session state, and human experience already exist inside the page. Routing media and transcripts through another server just so an agent can understand them would add latency, infrastructure, and privacy exposure.
We also learned that human-agent collaboration does not always mean asking an agent to operate a website. Hear My Site enables co-attention: the human and agent can follow the same live event, while the human controls when listening starts, which sources are included, and when the agent should respond.
Designing for multiple providers taught us that a reusable AI capability needs two kinds of stability: a semantic interface for agents and a transport abstraction for rapidly evolving AI services.
WebMCP tools should describe what the user wants to accomplish—start listening, retrieve new context, and stop listening—not expose the protocol details of a particular transcription provider.
Finally, a large text dump is not enough for live agent context. Stable cursors, bounded results, explicit lifecycle states, abort handling, provider-independent errors, and deterministic cleanup are what make continuous transcription usable as an agent capability.
What's next for Hear My Site
The hear-my-site 0.0.1 package is now available on npm. Next, we want to integrate more live transcription providers behind the same adapter interface, expanding provider choice without changing the agent-facing contract.
We also want to publish complete, practical integration guides for WebRTC meetings, live media players, and other audio-rich websites. These guides will cover supplying browser-owned media tracks, using site-minted ephemeral credentials, choosing product-specific WebMCP tool names, and keeping transcript state inside each page.
We also want to explore broader browser support, optional TURN or SFU integrations for larger and less reliable meeting networks, and richer transcript metadata when providers offer reliable speaker attribution.
Our larger goal is to make agent-readable live audio a native capability of the open web: websites provide the experience, users retain control, providers remain interchangeable, and agents can finally understand what their users are hearing.
Log in or sign up for Devpost to join the conversation.