oa::Network - WebMCP for native Vulkan applications

Safe, typed WebMCP tools for native C++ Vulkan, ML, and 3D applications.

Comment

Inspiration

AI agents can operate websites, while serious compute, ML, media, and 3D applications still live behind native C++ runtimes and GPU APIs. Existing browser automation often forces agents to guess through pixels and UI controls, while conventional MCP servers operate separately from the live page the user is viewing.

OA already provides a native Vulkan compute and ML runtime, a bounded MCP implementation, training sessions, rendering, video, and experimental prompt-to-motion generation. We wanted to connect those capabilities directly to WebMCP so a person and an agent can safely operate the same live native application—with OA remaining the execution engine rather than moving the workload into JavaScript or a hosted service.

What it does

oa::Network - WebMCP exposes validated OA application actions as page-local WebMCP tools.

A native C++ OA process owns the Vulkan device, compute graphs, models, rendering resources, synchronization, and application state. A small TypeScript adapter discovers its registered tools, exposes them through WebMCP, and forwards calls to the same typed commands used by the human interface.

The project is designed around two demonstrations:

  • A tiny-language-model workspace for selecting a model and tokenizer, starting or pausing training, inspecting bounded metrics, evaluating, checkpointing, and generating text on a local Vulkan GPU.
  • A prompt-to-motion workspace using OA’s experimental ALM stack, native rendering, timeline state, and deterministic motion-editing tools.

Large tensors, model weights, video frames, and GPU handles never travel through WebMCP. The browser carries only bounded commands, status, metrics, identifiers, and verifiable results.

How we built it

The native implementation is built in C++20 as part of OA Network. oa::McpServer owns the immutable tool registry, JSON schemas, validation, bounded parsing, and application handlers. The WebMCP bridge composes that server through an explicitly owned network gateway rather than creating a second tool implementation.

The browser integration is a small TypeScript layer that:

  • discovers compatible OA tools;
  • registers them with document.modelContext.registerTool;
  • maps WebMCP annotations and JSON schemas;
  • forwards validated calls to the native process;
  • preserves cancellation, revisions, and ordered results;
  • updates the same React interface used by the human operator.

OA’s Vulkan engine performs the actual ML, rendering, and media work. The browser remains a control and presentation surface. Generic interface components come from our existing Realm UI package, while application state and WebMCP registrations remain owned by the OA SDK demonstration.

Challenges we ran into

MCP and WebMCP are related but have different ownership and lifecycle models. An MCP server can operate independently, while WebMCP tools belong to an open browser page and disappear when that page closes or navigates away. Sharing tool semantics without accidentally creating two canonical execution paths required a careful composition boundary.

Native GPU work is asynchronous and stateful. A WebMCP call being accepted does not mean a training mutation, checkpoint, render, or generation request has already completed. We needed explicit accepted, applied, completed, rejected, cancelled, and failed states.

Security was another central challenge. The bridge must reject oversized or malformed input, arbitrary filesystem access, URLs, shader source, Vulkan handles, and unbounded allocation requests. Local execution also requires strict loopback binding, origin validation, per-launch credentials, bounded connections, and predictable shutdown.

Finally, we needed to keep the browser responsive while the native Vulkan application continued running independently of page refreshes, reconnects, or agent availability.

Accomplishments that we're proud of

OA already provides a transport-independent MCP server with strict JSON parsing, bounded messages, deterministic tool registration, typed arguments, structured results, and adversarial tests.

Its training adapter uses the existing oa::TrainingSession safe-point command system rather than mutating model or GPU state directly. The same session can therefore be controlled by native UI actions, MCP, and WebMCP without changing the underlying training semantics.

We are particularly proud that OA remains the product:

  • compute and rendering execute through our native Vulkan runtime;
  • one oa::Engine owns GPU resources and scheduling;
  • JavaScript does not emulate or replace the native workload;
  • tool calls expose semantic actions rather than raw implementation handles;
  • the human and agent work with the same live state;
  • WebMCP becomes a reusable OA Network capability rather than challenge-specific glue.

What we learned

WebMCP is most useful when it exposes meaningful application operations instead of reproducing every button in the interface.

The strongest human-agent experiences share visible state, history, selection, and results. The agent should be able to act precisely, while the user can immediately inspect what changed and undo or refine it.

We also learned that control traffic and bulk data need separate paths. JSON and WebMCP are appropriate for commands, metrics, provenance, and resource identifiers—not tensors, model files, video frames, or GPU memory.

Most importantly, a generic bridge between a validated native tool registry and WebMCP is more valuable than writing isolated JavaScript handlers for every demonstration.

What's next for oa::Network - WebMCP for native Vulkan applications

Our first milestone is the complete tiny-language-model demonstration: live Vulkan training, metrics, pause and resume, evaluation, checkpoint reload, generation, and WebMCP control through one shared browser workspace.

If time permits, the second demonstration will add prompt-to-motion generation, native 3D preview, timeline inspection, deterministic edits, and export through OA’s ALM and rendering stack.

After the challenge, we plan to:

  • qualify the bridge under sanitizers and adversarial protocol tests;
  • add installed-package and fresh-browser testing;
  • expand reusable TypeScript bindings without moving compute into the browser;
  • support additional OA vision, audio, video, and 3D applications;
  • benchmark the complete native-to-browser control path;
  • add authenticated remote deployment while preserving the secure local default.

Built With

Share this project:

Updates