Auralis WebMCP

A WebMCP-powered human-agent listening experience that illustrates how hearing loss and competing sounds can make family conversations harder to follow—while keeping the person in control.

Comment

Inspiration

I work as an audiology technician, and every day I encounter not only hearing loss itself, but also the communication gap that forms around it.

An audiogram can show numbers and curves. What is much harder is helping a family understand why a conversation with the television on can be exhausting, why the speaker’s position matters, or why someone may hear that speech is present without understanding it clearly.

The original Auralis was therefore created as an illustrative, non-clinical listening experience. It lets people compare the same family scene under different synthetic listening conditions. It is not intended to diagnose hearing or reproduce any individual person’s perception exactly. Its purpose is to create a shared and understandable starting point for conversation.

WebMCP took that idea much further. What began as a human-controlled illustration became a shared live space where a person and an agent collaborate over the same scene, the same state, and the same safety boundaries.

What it does

Auralis begins as a normal human-first application. Simply opening, browsing, or clicking the page never activates agent mode.

The first successful stage_demo_session call transforms the full presentation into Agent Reveal. This is not merely an agent clicking through the existing website. The page gains a new information hierarchy, session progress, controls, and a final summary designed specifically for a shared human-agent journey.

The person and the agent are not operating two separate applications. Both work through the same revisioned ExperienceState.

A typical session works like this:

  • The agent explains the illustrative, non-diagnostic purpose.
  • The person must confirm consent directly.
  • The agent introduces three approved synthetic profiles and may recommend the one with the clearest demonstration contrast.
  • The person can override the choice by clicking or by responding to the agent through text or voice, where the current host and phase support it.
  • Auralis automatically prepares the fixed family scene.
  • The person visibly confirms a safe and comfortable listening level.
  • They may separately grant the agent permission to start audio.
  • The agent prepares and starts the reference A and illustrative B listening comparison.
  • The person describes what they noticed in ordinary language.
  • The agent translates only that stated meaning into bounded, non-diagnostic fields.
  • It can then change one approved variable at a time—for example, turning off the television, adding illustrative left-ear support, or changing the speaker’s position when explicitly requested.
  • The session ends with a deterministic summary of what was compared and what the person actually reported.

The person retains consequential authority throughout the experience. WebMCP cannot provide consent or acknowledge physical listening volume on their behalf. Agent-started playback requires a separate grant, and Stop remains immediately available. Interrupted audio never counts as heard.

Two defining WebMCP features

1. Agent Reveal and a real transformation of the experience

The first successful stage_demo_session call transforms the complete page presentation into Agent Reveal.

Later WebMCP calls change only approved listening variables through the shared ExperienceState. The agent can therefore substantially change the flow and presentation of the experience without receiving arbitrary DOM control or bypassing the safety model.

A direct human click is also a real event in the same session. The agent does not need to imitate that action through Computer Use.

2. A continuing session that can wait for the person

One of the hardest problems was preventing the agent from treating every human step as the end of the session and waiting for an entirely new prompt.

I built wait_for_session_change as a bounded, read-only call tied to the exact session and revision. Normal human or technical steps use a four-minute window, while the optional profile override deliberately uses a shorter 15-second window.

A bounded wait_for_session_change call can keep the current agent run pending while the host keeps the tool call active. When the page revision changes, the call returns the new semantic state and the agent can continue with the next allowed step without model polling.

Direct page actions and technical page progress resolve the wait. Typed or spoken responses are handled by the compatible host, which can then invoke the next allowed WebMCP tool.

This is not a WebSocket, durable backend persistence, or a mechanism that can revive a turn the host has already ended. It is a practical way to keep an active human-agent interaction connected through ordinary WebMCP calls.

How I built it

Auralis uses:

  • Next.js, React, and TypeScript;
  • the Web Audio API for deterministic in-browser audio processing and playback;
  • Zod for strict input and output contracts;
  • WebMCP for the semantic connection between the page and a compatible agent;
  • Vitest for domain and browser-adapter testing;
  • Playwright for complete browser journeys.

The WebMCP extension exposes exactly seven narrowly scoped tools:

  • get_auralis_state
  • stage_demo_session
  • prepare_comparison
  • play_listening_comparison
  • record_user_observation
  • get_session_evidence
  • wait_for_session_change

allowedNextTools is the sole authority for what may execute in the current phase. Every mutation is bound to an expected revision and protected against stale, repeated, or conflicting requests.

Auralis does not expose raw audiogram thresholds or free-text observations through WebMCP. Session evidence contains only a bounded, ordered, and attributable record.

Auralis’s server-side OpenAI explanation layer is optional and never controls the audio. Agent Reveal itself requires a compatible external WebMCP host and agent. The complete deterministic human journey remains available without the internal OpenAI explanation layer and when WebMCP is unavailable.

Challenges I ran into

  • Keeping one truthful state for both direct human actions and agent tools.
  • Allowing the agent to wait without polling or pretending that the page has durable backend persistence.
  • Ensuring that the agent continues correctly after a human decision.
  • Separating the full visual Agent Reveal from the agent’s deliberately bounded authority over listening variables.
  • Keeping consent, physical listening volume, the playback grant, and Stop under human control.
  • Preventing the agent from speaking over the audio. The page therefore inserts an eight-second quiet lead-in before agent-started playback, and the tool remains pending until completion, failure, or a human Stop.
  • Translating ordinary human descriptions into structured observations without diagnosing or inventing meaning.
  • Preserving a complete human fallback for browsers without WebMCP.

There was also one technical challenge I did not manage to solve: the hackathon deadline has no Pause button. With a few more days, Auralis would probably already be explaining why I still have not slept.

Accomplishments that I’m proud of

  • WebMCP is not just a wrapper around clicking; it is a semantic part of the product.
  • The agent can activate a completely different presentation and guide the person through the full experience.
  • The person can intervene directly in the same session at any time.
  • After explicit permission, the agent genuinely starts the listening comparison instead of telling the person to press Play.
  • Audio, profiles, scene conditions, adaptations, observations, and the final summary remain consistent.
  • The original human Auralis journey remains intact, with the WebMCP work preserved as a traceable extension above it.
  • The repository includes domain, browser-adapter, and Playwright journey coverage; separate owner-observed sessions covered native typed and voice-guided use.

What I learned

The most important lesson was that a good agentic application needs more than a list of tools. It needs a clear continuation contract, explicit ownership of state, and an exact answer to who is allowed to perform the next action.

I also learned that waiting is not merely a technical detail. In a human-agent product, it is part of the user experience. The agent must understand when to act, when to explain, when to remain silent, and when to genuinely wait for the person.

WebMCP made it possible to keep the human as an active participant and owner of consequential decisions, rather than reducing them to a spectator of automation.

What’s next for Auralis

I want to develop Auralis into a broader educational platform focused on hearing loss and the available approaches to its correction and support.

Future directions include:

  • more everyday listening situations;
  • more illustrative profiles and listening conditions;
  • accessible explanations of communication strategies, hearing aids, and other assistive technologies;
  • better preparation for a professional hearing consultation;
  • decision support that helps people compare options more thoughtfully without replacing diagnosis, device fitting, or the judgment of a qualified professional;
  • more accessible education for families, partners, and people encountering hearing loss for the first time.

Today, Auralis illustrates one family scene. The longer-term goal is to help people not only see hearing loss on an audiogram, but understand how it can affect real life—and what can safely and meaningfully be done about it.

I wrote the original project story in Czech and used AI assistance to translate and refine it, aiming to preserve my meaning as clearly and accurately as possible in English.

Built With

Share this project:

Updates