AetherUI

A visual protocol compiler that turns any human web interface into structured MCP tools, no native API needed.

Comment

Inspiration

At Intel Foundry, I've been building an agentic assistant to help resolve customer-facing issues faster. On paper it's simple: understand the issue, gather the facts, propose a fix. In practice, the facts live everywhere. They're in lot trackers, product roadmap and release portals, internal dashboards, and custom tools that have quietly done their job for years. Most of them have one thing in common: a web page, and no API.

So the agent hit a wall. Every answer it needed was one click away, and it couldn't click.

Meanwhile, every week brings a new AI agent demo that books flights and writes emails. Then it stops dead at exactly the place real work begins.

Building an API for each tool isn't quick, even for modern platforms that already ship one. In a foundry, connecting an agent to any system means a dedicated integration, a security review and an approval, and that's exactly how it should be. Security comes first: customer data must never cross between customers, and one wrong action by an agent improvising in a production system can become a real incident. Letting a free-roaming browser agent click around was never an option.

That left three real problems:

  • Waiting: weeks or months for each integration to be built and approved.
  • Trust: agents need a narrow, predictable set of actions, not an open browser.
  • Effort: every new MCP server means someone has to work out what to build, then build and maintain it.

So I asked a different question: what if an agent could learn a website once, the way a person does, and then use it like an API, with proof of every action it takes?

That's AetherUI.

What it does

Paste a URL. Get an MCP server.

AetherUI looks at a website once and compiles it into an MCP manifest: a JSON file of typed, validated tools. One connector serves every compiled site to Claude Code, Claude Desktop or Cursor.

On our demo legacy ERP, FabTrack (no public API), Claude instantly gets tools like:

  • fabtrack_query_wafer_lot_telemetry(status_filter): READ
  • fabtrack_execute_wafer_lot_action(lot_id, action_type, authorized_engineer): WRITE. It creates the lot if it's new, otherwise updates it.

Then you just ask: "Quarantine LOT-8812-kr and confirm the change." Claude calls the tool. The record changes. Verified.

Every call is:

  • Deterministic: the agent can only do what the compiled tool allows. No free-roaming clicks and no surprise actions, which is exactly what security-sensitive environments need.
  • Self-healing: when a surprise "ERR-902 compliance" popup blocks the page mid-transaction, Nemotron spots it, picks the right button, and finishes the job.
  • Network-first: reads return the records the site's own backend sent, not scraped text. On YouTube, ads simply drop out.
  • Verified: after every write it checks that the change really happened, and it reports partial success honestly.
  • Provable: a screenshot timeline, an execution trace, and the network endpoints used. Receipts for everything.

How we built it

  • The compiler, NVIDIA Nemotron on Nebius Token Factory. Playwright opens the page and stamps a numbered badge on every interactive element (Set-of-Marks). Nemotron-3.5-Super-VL reads the screenshot, the element index and the page text, and proposes typed MCP tools. A validator then grounds every step against the live page: invented steps are dropped, dropdown values are copied from the real <select>, and stable selectors are pinned. The model proposes; the page decides.
  • Vision where it matters, not everywhere. Most browser agents send a screenshot to a vision model on every click, which is slow, expensive and unpredictable. AetherUI calls Nemotron once to compile, and then only when something unexpected happens: a moved element, a blocking popup, or the final verification. Replay is deterministic. It's a more elegant use of a vision model: intelligence for understanding and recovery, determinism for execution.
  • The network-first reader listens to the site's own JSON traffic, filters out telemetry and ads, and finds the real records generically, with no per-site code.
  • MCP: FastMCP turns every manifest into typed tools, prefixed per site, so a whole company's portals can share one connector.
  • Built to survive: every model call has a fallback, text-only models are detected automatically, and a startup check tells you exactly what's misconfigured.
  • Ships like a product: setup.sh, run.sh, connect.sh, a .env file and an MIT license. Clone to working in three commands.

Challenges we ran into

  • Grounding, not generating. Early compiles invented steps, like typing into a button or using elements that didn't exist. The breakthrough was treating the model's output as a proposal and validating every step against the live page. That's what makes the tools trustworthy.
  • Scraping isn't truth. Reading YouTube's visible text gave us ads, Shorts and menu labels. Switching to the site's own network responses fixed that, and taught us that the biggest record family isn't always the right one, so we rank by richness.
  • Agents take tool descriptions literally. Claude refused to create a lot because the compiled description said the tool acts on existing lots. We taught the compiler to describe create-vs-update behaviour and which fields can't be set. Precise descriptions changed agent behaviour more than any prompt did.

Accomplishments that we're proud of

  • We took the MCP server out of MCP. No writing, no designing, no working out what to build. Paste a URL and the tools exist. What used to take a developer days (understanding the site, designing the schema, coding, testing) now takes seconds.
  • A coding agent operating a system with no API. Claude Code, inside VS Code, creates, updates and queries records in a legacy portal through typed tools nobody wrote.
  • Safety by design. Agents get a small, typed, deterministic surface instead of a free-roaming browser. Every action is verified and every result comes with receipts.
  • One connector for a company's entire web. Compile ten internal portals, and every agent can use all of them through a single MCP connection. Developers get it in their IDE; non-developers just ask in plain English.
  • Instant POCs and MVPs. Want to know whether an agent can automate a workflow on some site? Compile it and try it in minutes, not sprints.
  • Honest by default: partial success is reported as partial, and every answer states its data source.

What we learned

  • For agents, the tool description is the interface. Make it precise, and the agent behaves.
  • Vision is a superpower when used sparingly: to understand once, and to recover when the world changes. Not on every click.
  • A browser has to receive data in order to show it, so the network layer is often the best API a website already has.

What's next for AetherUI

AetherUI is the first step toward a bigger idea: a universal, trustworthy interface layer between AI agents and all the software humans use.

  1. From UI to real APIs, automatically. AetherUI already observes the network calls behind every action. Next, it promotes them into direct, fast API tools, effectively discovering the hidden API inside every web app, with the browser kept as fallback and proof.
  2. An enterprise agent gateway. Login sessions, role-based permissions, approvals for writes and full audit logs, so security teams can approve agent access to internal systems in days instead of quarters.
  3. Powering agentic browsing at scale. Platforms like Tavily, which already give agents web access, could use compiled manifests to turn pages agents visit into tools agents can use: from "read this page" to "act on this page, safely and repeatably." (We haven't integrated Tavily yet; this is where we see a natural fit.)
  4. A shared library of compiled sites. Manifests are portable JSON, so teams and communities can publish and reuse them, the way package registries made libraries reusable.
  5. Beyond the browser. The same compile-and-validate approach extends to desktop apps through their scripting APIs and accessibility trees.
  6. Hosted on Nebius AI Cloud. Connect with one URL, no install.

Known limits, stated honestly: a tool can only do what the UI allows, compiles vary slightly between runs (every tool is still grounded on the real page), there's no login support yet, and calls take about 6–15 s. CAPTCHAs and site terms are respected, never bypassed.

Built With

  • agent
  • chromium
  • claude-code
  • fastapi
  • fastmcp
  • html
  • javascript
  • mcp
  • model-context-protocol
  • nebius
  • nebius-token-factory
  • nemotron
  • nvidia-nemotron
  • openai-api
  • playwright
  • pydantic
  • python
  • set-of-marks
  • tailwindcss
  • uvicorn
  • vision-language-models
Share this project:

Updates

Submission history