Inspiration
WebMCP gives websites a semantic interface for agents. But what about the enormous number of applications that were never built to expose one?
Most browser automation approaches teach agents where to click or how to reproduce a sequence of UI actions. That makes the agent depend on interface mechanics rather than application meaning.
We asked a different question:
What if a user could teach an application a capability simply by demonstrating how they already perform the task?
AutoWebMCP turns those demonstrations into agent-ready WebMCP tools.
Teach → Semanticize → Validate → Publish → Agent
The key idea is simple: the demonstration is evidence, not the automation.
What it does
A user performs a normal workflow while the AutoWebMCP Teach Mode extension observes the interaction.
AutoWebMCP analyzes that evidence and proposes a typed semantic capability representing what the user intended, rather than exposing individual clicks as agent actions.
For example, a human editing several controls in Salesforce can become:
update_opportunity_details(opportunity_id, stage, close_date, main_sponsor)
The user reviews and confirms the semantic contract. AutoWebMCP then establishes an execution binding, validates it against the live application, and publishes the capability as a WebMCP tool.
The resulting pipeline is:
Human demonstration → Semantic capability → Execution binding → WebMCP tool → Agent
We tested this with two very different applications.
SignalBase is a cooperative application. AutoWebMCP can bind a learned capability to an action supplied by the application itself.
Salesforce Lightning is the harder case. It was not built for AutoWebMCP, so learned capabilities execute against the application's existing interface through the AutoWebMCP runtime.
To an agent, both appear as semantic WebMCP tools with names, descriptions, typed schemas, identity requirements, and safety semantics.
In our demo, one natural-language instruction composes three independently taught capabilities:
- Find a VP-level contact in SignalBase.
- Search Salesforce for the relevant Opportunity and obtain its exact record identity.
- Update that Opportunity.
The sequence is not hardcoded. The model chooses among the live WebMCP tools using their published descriptions, schemas, and results.
How we built it
AutoWebMCP has three main parts.
Teach Mode is a Chrome extension that captures evidence while a user performs a workflow. It observes the interaction without treating the resulting event sequence as a replay script.
Training Studio turns that evidence into a semantic capability. A model proposes the business intent, capability name, description, and typed inputs. The user reviews and confirms that contract before publication.
The demonstration alone is not enough to safely execute the capability. AutoWebMCP combines observed evidence with platform, application, and tenant intelligence to help ground concepts such as business meaning, valid value domains, entity identity, tenant-specific vocabulary, execution semantics, and verification behavior.
Critically, the semantic contract remains separate from the execution binding. An agent operates on concepts such as stage, close_date, and opportunity_id, while the runtime resolves how those concepts map to the application when the capability executes. We do not persist CSS selectors, XPath expressions, or screen coordinates as the semantic contract.
Finally, WebMCP is the publication surface. Confirmed capabilities are compiled into tools with descriptions, typed input schemas, required fields, value domains where known, and supported annotations.
For the cross-application demonstration, we also built a small page-side agent harness over the actual WebMCP API. On every step it:
- reads the current tools from
document.modelContext.getTools() - gives those descriptions and schemas to the model
- validates the selected tool and arguments against the live tool set
- invokes the tool through
document.modelContext.executeTool() - feeds the structured result back to the model for its next decision
There is no separately hardcoded SignalBase-to-Salesforce workflow. The model has to infer the composition from the semantic tools it is given.
Challenges we ran into
The hardest problem turned out not to be recording browser interactions. It was deciding when an inferred capability is trustworthy enough for an agent to use.
A demonstration contains incomplete and sometimes ambiguous evidence. A field label may be tenant-specific. A choice may have a constrained value domain. Two records can have similar names. A successful request does not necessarily mean the intended state was persisted.
That pushed AutoWebMCP toward a stronger principle:
Search may be fuzzy. Mutation must be exact.
Writes therefore require an exact target identity, and execution verifies both the target and the requested outcome. If the runtime cannot establish what happened, it reports uncertainty instead of manufacturing success or blindly repeating a potentially committed mutation.
Testing against a live Salesforce org exposed failure modes we would not have found in a controlled demo. Navigation can destroy the browser context waiting for an execution result. Browser extension service workers can disappear between captured events. A mutation can succeed even when its caller never receives the result.
Those failures shaped the runtime's guardrails, retry semantics, target-identity checks, and distinction between requested state and verified state.
Cross-application composition created another challenge. WebMCP tools belong to documents, while our demo combines a capability taught from SignalBase with capabilities backed by Salesforce. We therefore made publication location explicit and created a control surface where selected capabilities can coexist while still being invoked through the WebMCP API.
Accomplishments that we're proud of
The biggest accomplishment is that the original idea survived contact with a real enterprise application.
We demonstrated the full loop:
demonstrate → semanticize → human confirm → bind → live validate → publish → discover → compose → execute → verify
The same learning pipeline supports two very different execution strategies: a cooperative application-provided binding in SignalBase and a learned browser execution binding against Salesforce Lightning.
We are particularly proud that the agent-facing abstraction is semantic rather than mechanical. The agent sees:
update_opportunity_details(...)
not:
click_button(...) → fill_input(...) → choose_option(...) → click_save(...)
We also demonstrated model-driven composition across independently taught capabilities. From one business instruction, the agent selected a SignalBase contact capability, a Salesforce Opportunity search capability, and a Salesforce update capability based on the published WebMCP metadata and prior tool results.
The safety behavior became an accomplishment of its own. AutoWebMCP can refuse to mutate an entity it cannot identify, distinguish an attempted change from a verified outcome, stop when a write may already have persisted, and preserve ambiguity rather than silently choosing among multiple candidate records.
Those are not theoretical safeguards. Several were added after the corresponding failure happened during live testing.
What we learned
Our biggest learning is that turning a demonstration into an agent capability is fundamentally different from recording a macro.
The interaction trace tells us what happened. It does not, by itself, tell us what the action means.
The difficult work lies in extracting a durable semantic contract from imperfect evidence, grounding that contract in the application's concepts and environment, and then resolving it safely when an agent invokes the capability later.
We also learned that semantic understanding and execution should remain separate. A stable concept such as close_date should not become synonymous with whatever DOM element happened to represent Close Date during the demonstration.
And we learned that verification is part of the capability, not an afterthought. For autonomous systems, "I attempted the action" and "I established that the intended state now exists" are materially different outcomes.
This changed how we think about AutoWebMCP. What began as an experiment in automatically generating WebMCP tools increasingly looks like a semantic capability compiler.
What's next for AutoWebMCP: Teach an Agent to Fish
WebMCP is the first publication target, but the learned semantic capability does not inherently belong to WebMCP.
The same architecture could eventually publish a confirmed capability through WebMCP, an external MCP server, an API, or another agent runtime while preserving the semantic contract and its safety requirements.
We also want to extend the capability model beyond updates and searches to creation, richer entity relationships, reference fields, temporal concepts, and more applications.
Longer term, the interesting question is broader than browser automation:
Can software learn its agent interface from the people who already know how to use it?
WebMCP gives cooperative websites a native way to expose semantic tools.
AutoWebMCP explores how to bring that same idea to the applications that already exist.
Teach the workflow. Semanticize the intent. Give the agent the capability.
Or, simply:
Teach an Agent to Fish.
Built With
- ai-agents
- browser-automation
- chrome
- core
- css
- html
- javascript
- node.js
- openai
- openai-api
- salesforce
- salesforce-lightning
- semantic-web
- typescript
- vite
- webmcp
Log in or sign up for Devpost to join the conversation.