Esbiko WebMCP Science Lab

Esbiko turns interactive science simulations into shared workspaces where people and AI agents configure, run, inspect, and compare real experiments through WebMCP.

Comment

Inspiration

Interactive science simulations are designed for people: learners move sliders, press buttons, watch animations, listen to sounds, and interpret measurements.

For an AI agent, however, the same interface is ambiguous. It has to locate controls visually, infer their scientific meaning, approximate numerical values, determine whether an experiment actually ran, and scrape the result back from the page.

I asked a different question:

What if a human and an AI agent could operate the same scientific experiment together?

Esbiko is an existing open-source science and mathematics learning platform with browser-based 2D and 3D simulations, classroom workflows, scientific visualisations, and media tools.

For the OpenAI WebMCP Challenge, I extended Esbiko with a semantic agent-facing layer that makes its Doppler Effect laboratory directly understandable and controllable through WebMCP.

The goal is not to replace the learner or hide the experiment behind a chatbot. The human should still see, hear, inspect, and modify the experiment. WebMCP gives the agent a precise way to act on that same scientific environment.

The result is a shared scientific workspace where human intent, agent actions, visible simulation state, physics calculations, audio, and recorded output all remain connected.

What it does

Esbiko WebMCP Science Lab turns a live Doppler Effect simulation into an agent-operable scientific instrument.

The current implementation exposes 11 semantic WebMCP tools.

Two site-level tools allow an agent to discover and open WebMCP-enabled science simulations:

  • list_science_simulations
  • open_science_simulation

Nine Doppler tools allow the agent to read and control the experiment:

  • get_doppler_state
  • configure_doppler
  • configure_doppler_scene
  • set_doppler_playback
  • reset_doppler
  • create_doppler_video
  • get_doppler_video_status
  • stop_doppler_video
  • download_doppler_video

An agent can configure source motion, source speed, emitted frequency, observer state, direction, sound type, and multi-source scenes.

It can also direct a complete scientific video.

The final production demonstration creates a 30-second, 9:16 two-vehicle Doppler experiment at an emitted frequency of 440 Hz and a source speed of 60 m/s.

A Real Car Engine travels from left to right. It approaches the stationary observer, passes at 7.5 seconds, and then recedes.

An Ambulance Siren then approaches from the opposite direction, passes the observer at 22.5 seconds, and continues away.

For a speed of sound of 343 m/s, Esbiko calculates approximately:

533.29 Hz while approaching

and

374.49 Hz while receding.

The pitch difference is clearly audible in the generated video.

While the agent is recording, the user can also watch the same directed motion happening in the live browser.

After recording, the agent can inspect structured recording status, confirm that real audio was captured, wait until the file is ready, and download the final WebM.

How I built it

Esbiko uses the WebMCP Imperative API through document.modelContext.registerTool(...).

The implementation is a progressive enhancement over the existing React application. Browsers without WebMCP continue to use Esbiko normally.

Two persistent site-level tools provide discovery and navigation. Nine Doppler-specific tools are registered while the Doppler laboratory is active.

A validated JSON-safe adapter connects WebMCP inputs to Esbiko's existing React state and scientific code.

Semantic instructions such as approaching, receding, and explicit source directions are converted into validated physical parameters instead of arbitrary DOM interactions.

Scientific results are never generated by an LLM. Esbiko's application code deterministically calculates values such as:

  • Observed frequency
  • Frequency ratio
  • Percentage frequency shift
  • Relative amplitude
  • Sound level
  • Motion classification

The AI video workflow required an additional Doppler Director.

The director creates an exact timeline from physical parameters using:

distance = speed × time

For the final 30-second story, each phase lasts 7.5 seconds and each vehicle travels 450 metres during each phase.

The observer remains fixed at 500 metres.

The recorder creates a deterministic 1080×1920 WebM and combines the rendered simulation with a real Web Audio track.

Selected sound samples are preloaded before recording. Esbiko routes the Web Audio mix through an analyser and a MediaStreamDestination.

Before accepting a recording, Esbiko checks that a measurable audio signal actually reaches the recording bus. A silent track is not treated as success.

Most importantly, during AI-directed recording, the live browser motion, recorded motion, and Doppler audio are driven from the same deterministic director timeline.

This keeps the audible pitch transition synchronized with the exact visual moment when each vehicle passes the observer.

Challenges I ran into

One of the hardest problems was not implementing the Doppler equation itself. It was keeping several real-time systems synchronized.

Initially, the recorded video used a deterministic director clock while the sound was still driven by the ordinary browser simulation loop.

Under the additional workload of rendering a 1080×1920 canvas and encoding WebM, browser frame timing could slow down.

The vehicle in the video would reach the observer at the correct time, but the audio simulation could lag behind it. The Doppler shift was scientifically correct, but the audible pitch change did not always line up clearly with the visual crossing.

I eventually removed that split-clock architecture.

During AI recording, the audio source state is now derived directly from the same deterministic director timeline used by the live browser preview and recorded video.

That made the higher-to-lower pitch transition clearly audible at the intended pass moments.

Browser audio policy was another challenge. Modern browsers may prevent an AudioContext from starting without a human gesture. Esbiko preserves that browser security model and returns an explicit AUDIO_ACTIVATION_REQUIRED error when necessary.

I also had to solve silent recording failures. Having an audio track attached to MediaRecorder does not guarantee that meaningful sound exists in it, so Esbiko performs a real analyser-based signal test before starting the recording.

Finally, because Esbiko is an existing Progressive Web App, browser caching and Service Worker state had to be considered during production verification to ensure I was testing the current deployed revision.

Accomplishments that I'm proud of

I built an end-to-end WebMCP workflow inside a real existing educational application rather than creating an isolated WebMCP technical demo.

The production system now demonstrates semantic discovery, navigation, scientific configuration, scene control, playback, physics-derived measurements, AI-directed recording, real audio verification, recording status, and downloadable scientific media.

The final production test successfully demonstrated a 30-second two-vehicle scenario with a Real Car Engine and Ambulance Siren moving in opposite directions at 60 m/s.

Both vehicles visibly move in the live browser while the recording is being created.

The Real Car Engine passes the observer at approximately 7.5 seconds and the Ambulance passes from the opposite direction at approximately 22.5 seconds.

In both cases, the transition from higher approaching pitch to lower receding pitch is clearly audible and synchronized with the visual observer crossing.

Esbiko also supports a simpler 10-second single_pass mode where one continuous siren approaches for five seconds and recedes for five seconds, providing an especially clear A/B Doppler comparison.

The implementation includes structured validation, bounded scientific inputs, explicit browser-audio errors, audio-signal verification, deterministic recording, shared human/agent state, and automated WebMCP contract and physics tests.

The complete implementation, architecture, testing documentation, challenge evidence, deployment history, and MIT-licensed source code are public.

What I learned

I learned that the most useful WebMCP tools represent human intent, rather than mechanically exposing every UI control.

A scientific user does not really want to say:

"Move slider number three to pixel position 240."

They want to say:

"Show me a 440 Hz source approaching an observer at 60 metres per second."

WebMCP provides a much better boundary for expressing that intent.

I also learned that adding agent control makes the visible human interface more important, not less.

The strongest workflow is not one where the AI operates invisibly in the background. It is one where the agent performs precise and repeatable operations while the learner can simultaneously see, hear, inspect, question, and modify the experiment.

For scientific software, the agent should not become the scientific source of truth.

Physics equations, parameter bounds, measurements, validation, and experiment state should remain deterministic and inspectable application logic.

The AI helps operate the scientific instrument.

It does not replace the instrument.

What's next for Esbiko WebMCP Science Lab

The Doppler experiment is the first fully developed WebMCP reference implementation inside Esbiko.

The next step is not to expose every simulation immediately. I want to apply the same pattern carefully to additional Esbiko experiments, with small semantic tool surfaces and scientifically validated adapters.

Future WebMCP-enabled workflows could include orbital mechanics, satellite systems, waves, optics, electromagnetic fields, and other interactive experiments already represented in Esbiko.

Longer-term possibilities include:

  • Teacher-created experiment presets
  • Repeatable experiment records
  • Guided comparisons between scientific scenarios
  • Student prediction-and-observation activities
  • Agent-assisted lesson preparation
  • Classroom review workflows
  • Automatic production of educational scientific media

The broader vision is an open science-learning environment where people and agents can investigate the same visible world together — while preserving human curiosity, scientific transparency, and human control.

Built With

Share this project:

Updates