InnerOS WebMCP Mission Control

A WebMCP coding control plane where a local AI proposes changes, InnerOS routes execution across real agent lanes, and every backend hop returns verifiable proof.

Comment

InnerOS WebMCP Mission Control

Inspiration

Most AI agents still interact with websites by looking at pixels, guessing what controls mean, and imitating human clicks. That is fragile for serious development, operations, and physical automation.

We wanted a different model: the website itself should expose governed capabilities to the agent, while the human keeps a visible control room and every meaningful action remains verifiable.

InnerOS already existed as our local-first multi-agent operating layer, coordinating local models, MCP/A2A agents, coding providers, infrastructure, evidence, and physical automation. For The WebMCP Challenge we built a WebMCP-native product layer that turns that infrastructure into one public, agent-operable development and operations cockpit.

Live judge URL: https://webmcp.creatorcore.ai

The core interaction rule

Conversation is deliberation. Approval is the execution boundary.

A user can talk to the local InnerOS Copilot for multiple turns, refine requirements, attach PDFs or source files, dictate a prompt by voice, change projects, and choose a different executor. None of that programs, registers, or physically executes anything.

Only when the user is satisfied do they press Approve & Execute Plan. InnerOS then validates the selected project again and dispatches the latest refined plan through the selected lane.

This gives Mission Control a clean universal workflow:

  1. Choose or create a project.
  2. Attach project context if useful.
  3. Converse with local Qwen and refine the plan.
  4. Choose AUTO / Local AMD / Codex / Cursor / AntiGravity.
  5. Press Approve & Execute Plan.
  6. Watch delivery, execution, verification, and evidence as different backend states.

Delivery is not execution. Registration is not physical execution. A dispatch ID proves acceptance, not completion.

A real browser development workspace

The challenge build now goes beyond a one-shot demo console.

Create Project

Mission Control has an explicit Create Project action. Typing a new name into the Project field alone never creates anything.

create_project_workspace creates a bounded local Git development workspace under the canonical InnerOS Projects root, initializes Git, and registers the project in Project Runtime. It deliberately does not create a GitHub or cloud repository behind the user's back.

Before any coding dispatch, InnerOS verifies that the selected project exists and is a real Git workspace.

PDF and source-code context

Authenticated users can attach bounded PDFs and common source/text files. Attachments are stored privately per project and are passed to Qwen as read-only planning context.

The Copilot now receives recent conversation history plus project-file context, so refinement is genuinely multi-turn rather than a chat UI that secretly sends only the latest message.

Text PDFs are extracted locally; source/text files are decoded directly. If a PDF is image-only and no text can be extracted, the UI says so instead of pretending the model read it.

A bounded portion of the same project context follows the execution brief after approval so the chosen executor can act on the relevant code/document information.

Local voice dictation

The voice button is local-first too.

The browser records microphone audio using MediaRecorder, sends it to the authenticated WebMCP origin, and the server forwards it over loopback to our already-running local Whisper ASR service. The transcript comes back into the chat composer for review.

Voice is dictation only. It never bypasses the approval gate.

If local Whisper is unavailable, browser speech recognition is offered only as an explicit fallback rather than being mislabeled as local.

Why WebMCP matters here

WebMCP is not decorative in this project. It is the machine-facing interaction model.

Instead of forcing an assistant to infer a human UI, the page publishes a compact set of semantic capabilities through document.modelContext.registerTool(...).

The current public surface has 13 WebMCP tools:

  1. ask_inneros_copilot
  2. list_agents
  3. get_project_status
  4. create_project_workspace
  5. inspect_blockers
  6. dispatch_agent_action
  7. resolve_project_blocker
  8. get_execution_trace
  9. get_evidence
  10. dmx_create_scene
  11. dmx_status
  12. dmx_set_scene
  13. dmx_blackout

The browser sees only those governed actions. InnerOS can coordinate a much larger private MCP/A2A fabric without exposing raw internal tools, private infrastructure, filesystem roots, model endpoints, or credentials.

Local-first AI and execution lanes

The primary Copilot is Qwen3-Coder 30B, served locally through vLLM on our AMD node. The live provider fabric exposes four coding lanes plus AUTO:

  • AUTO — after approval, InnerOS chooses the cheapest capable local-first route; a bounded native capability may be used directly when appropriate.
  • Local AMD / Qwen3-Coder — local vLLM + durable A2A/local execution, preferred under local-first policy.
  • Codex — headless-capable when the live provider fabric proves CLI/auth/process readiness.
  • Cursor — remote IDE/inbox when headless execution is not proven; delivery is not execution.
  • AntiGravity — remote IDE/inbox unless a real runnable session is proven; completion requires returned evidence.

Explicit lane selection remains explicit. If the user chooses Codex, Cursor, AntiGravity, or Local AMD, AUTO does not silently hijack that choice.

Physical execution with AG-59 DMX

The same control plane reaches the physical world through AG-59, the InnerOS DMX Stage Orchestrator.

A judge can describe a lighting behavior that does not exist, refine it in conversation, approve it, and then watch the local AI create the new governed capability.

After approval:

  1. Local Qwen designs one bounded declarative scene.
  2. Common aliases such as purple, blue, or disco ball are normalized into the canonical trusted vocabulary.
  3. AG-59 validates scene name, targets, colors, brightness, step duration, loops, and total duration.
  4. AG-59 atomically registers the scene in the live registry.
  5. Mission Control automatically discovers and selects the scene.
  6. The physical lights are still idle.
  7. The user separately presses Apply scene.
  8. Only then does dmx_set_scene drive the real fixtures.

Raw DMX channels, fixture addresses, private topology, and rapid full-stage strobe are never accepted from public/model input.

That separation is important: proposal != registration != physical execution.

Proof, not theater

Global Live Trace is designed around one question: how do I know this actually happened?

Browser intent and backend confirmation are visibly separate:

  • BROWSER events represent client-side intent and are not presented as backend proof.
  • BACKEND · CONFIRMED events come from real server responses.
  • Project creation is confirmed only after Project Runtime reports a real Git workspace.
  • A provider receiving a task is not shown as running until process/session evidence exists.
  • A DMX scene-registry change is confirmed only after AG-59 reports the new catalog.
  • Voice is labeled Local Whisper only after the loopback ASR service returns a transcript.

Every WebMCP request receives proof metadata including request ID, tool name, bridge identity, backend, timestamp, latency, whether execution is being claimed, and durable dispatch ID when applicable.

Safety and trust boundaries

The public browser never receives raw access to the private MCP runtime, local Qwen endpoint, local Whisper endpoint, physical DMX node, server paths, or credentials.

Key rules:

  • only allowlisted WebMCP operations are accepted;
  • direct MCP access remains loopback-only;
  • project names and local workspaces are bounded and verified;
  • attachment size/type/context are bounded and project-scoped;
  • private topology, paths, sessions, tokens, and credentials are sanitized;
  • the Copilot cannot claim execution;
  • conversation never executes;
  • delivery is not treated as running;
  • completion requires backend evidence;
  • higher-impact host/cloud changes remain behind audited approval gates;
  • DMX creation is declarative, bounded, revalidated, and separate from physical execution;
  • local-first routing is visible instead of silently escalating to paid external compute.

What is new for The WebMCP Challenge

InnerOS and its underlying local multi-agent infrastructure existed before the challenge. The challenge-period work is the WebMCP-native product layer that makes it directly usable from an agent-capable website.

Challenge-period work includes:

  • WebMCP registration with document.modelContext.registerTool(...);
  • the 13 public WebMCP contracts;
  • local Qwen3-Coder Copilot integration;
  • real multi-turn conversation context;
  • explicit approval-first execution semantics;
  • local Git project creation from Mission Control;
  • project-scoped PDF/source context;
  • local Whisper voice dictation;
  • judge-safe InnerOS bridge;
  • constrained loopback MCP integration;
  • canonical project/worktree binding for coding providers;
  • selectable execution lanes;
  • proof-oriented Global Live Trace;
  • request/backend/latency/dispatch evidence;
  • delivery-vs-execution state semantics;
  • Cloudflare public edge while keeping the origin local;
  • AG-59 DMX physical-control integration;
  • dynamic DMX scene discovery;
  • local-AI scene creation and governed registration;
  • challenge-specific unit and live smoke tests;
  • documentation separating pre-existing InnerOS capabilities from challenge work.

Technologies

WebMCP · JavaScript / Node.js · MCP · A2A · InnerOS · Qwen3-Coder 30B · vLLM · local AMD compute · Whisper ASR · Git · Codex · Cursor · AntiGravity · Cloudflare Tunnel / DNS · Cloudflare Worker edge attestation · Python · Art-Net / DMX

Current verification

The current application test suite reports 53 PASS, 0 FAIL, plus two explicit authenticated live-smoke skips when dedicated test credentials are not provided. Public health confirms 13 WebMCP tools, the MCP loopback bridge, local Qwen, AG-59 DMX, project creation, attachments, and voice-client capability.

We also live-validated the project-creation primitive by creating a real local Git sandbox under the canonical Projects workspace without creating a remote repository, and verified that the local Whisper service exposes the ASR endpoint used by Mission Control.

Recommended judge demo

  1. Open Mission Control and authenticate.
  2. Show the execution lanes, Coding Chat, Global Live Trace, project controls, file/voice controls, and DMX controls in one cockpit.
  3. Optionally create a fresh sandbox project with Create Project and show backend-confirmed Git workspace creation.
  4. Attach a small source file or text PDF, or dictate a request with Local voice.
  5. Ask Qwen for a change, then refine the plan once to demonstrate real conversation memory.
  6. Choose an execution target and press Approve & Execute Plan.
  7. Show the durable dispatch and trace/evidence without conflating delivery with completion.
  8. Then ask for a new DMX scene that does not exist, refine it, and approve the plan.
  9. Watch the new scene register and appear in the selector while the physical lights remain idle.
  10. Press Apply scene to execute it physically.
  11. Finish with Blackout.

13 WebMCP tools. Local Qwen3-Coder. Local Whisper. Project workspaces. Multiple execution lanes. Dynamic physical control. One governed control plane with proof.

Built With

Share this project:

Updates

Submission history