ShowMe

Where did they hide that setting? Your agent lights up where to click next, on your own screen. No help article, no guessing, no handing over your account. Next time you won't need to ask.

Comment

Inspiration

I'm bad at finding things in apps. Not proud of it, but it's true. Some real ones:

  • Auth0. I needed to change one setting for a login flow and spent way too long clicking around the dashboard before I found it. The docs said "go to Settings". There are a lot of places called Settings.
  • Jira. New team, new board, and a few days in I'm running standup from it in front of everyone, trying to find where you change an issue type or add a column while people wait.
  • Datadog. When I was learning it I couldn't find where to filter logs properly, and later I struggled to find where the RUM integrations live.

And that's me, with a technical background. I can only imagine how the websites regular people use every day make people from other backgrounds struggle.

These all have the same problem. The answer exists, someone wrote it down, and it still doesn't help, because it's not on your screen.

The AI answer to this is "let the agent do it for you". Agents are great, I use them all day. But that answer has two problems. You didn't learn anything, so next month you'll ask again. And the site now has to trust a robot with your login settings or your money.

And sometimes you just can't hand it off. You're presenting. You're running standup. You're the one who's supposed to know where it is. Nobody wants to say "hold on, let me ask the agent" in front of the room.

So I didn't want an agent that does it for me. I wanted one that points at my screen and says "there", waits for me to click, and next time I don't need it.

That's ShowMe.

What it does

You open a site in a browser that has WebMCP, and you ask your agent something like "how do I turn on international transfers?"

Then this happens:

  • The site gives the agent a map of its own interface. Every screen, every menu, every button, with a sentence about each one.
  • The agent works out the path and hands the whole thing to the page in one call.
  • The page lights up the first thing you need to click, with a short message. You click it. The next one lights up. And so on.
  • You don't wait for the model between steps. It's not involved between steps.
  • Click the wrong thing and the page tells the agent, which re-plans.
  • Say "just do it for me" and the site decides. Opening a menu, fine. Pressing Confirm on a transfer, no. The agent tells you why.
  • When you're done, the agent tells you the path in one line so you remember it.

I made two sites to show it. Neither has any guide code in it, just the attributes.

PeruBank is a fake bank. The switch for international transfers is four levels deep, and the final Confirm is a button I never want an agent pressing.

Meridian is a fake team tracker. Board settings behind one three-dots menu, project settings behind another, a type icon nobody realises is a button. Basically the board I struggled with.

Who it's for: anyone in an app with hundreds of settings, like developer dashboards and admin consoles. People who should never hand an agent their account, like my parents and their bank. And anyone learning a tool in front of other people.

Why WebMCP

To show someone where something is, you need three things, and only the page has them:

  1. A map of the interface that's actually true right now.
  2. A way to wait for a real click and know it happened.
  3. A say in what the agent is allowed to do.

A screen-reading agent guesses at all three. You could maybe get the map from an accessibility tree. You can't get the other two. WebMCP gives the site one standard way to hand all three to whatever agent the person is using. That's why this works now and didn't before.

How I built it

One JavaScript file, one stylesheet, no build step. Both sites use the same two files.

The site marks its elements:

  • data-guide is the id.
  • data-guide-desc is a sentence about it, written like you'd explain it to a new colleague.
  • data-guide-delegable means an agent may do this step for you. Leave it off and it's yours.
  • data-guide-danger means it can't be undone. Never delegated.
  • data-screen, data-panel, data-dialog, data-menu group things so the agent knows what has to be open first.

The map is rebuilt from the live page every time it's asked for, so anything rendered later is in it too. Whether something is open is read from what's actually visible on screen.

Five tools, all registered through document.modelContext.registerTool:

  • get_ui_map and get_current_view. Read-only. Flagged untrustedContentHint because they echo text people typed.
  • run_walkthrough. The long one. Takes the whole path, paces you through it on the page, listens for the AbortSignal, and comes back with what you did and where you are now. If it's taking a while it returns in_progress after 40 seconds and the agent calls again.
  • do_step_for_person. Reads the policy from the markup. Refuses anything not marked delegable.
  • end_walkthrough. Only exists while a guide is running. Registered with an AbortController, so when the guide ends the tool disappears and toolchange fires.

A Playwright test drives both sites through getTools and executeTool, with Google's polyfill injected so it runs on plain Chrome.

Each site also has an Agent activity panel. It shows which bits of the API have actually been used this session, plus a transcript of every call and every click. I added it because the agent's own call log is just two lines, and everything interesting happens inside one of them.

Challenges I ran into

The first version did one spotlight per tool call. In ChatGPT's browser every round trip was 10 to 20 seconds, so you'd click and then sit there. Handing the whole path to the page in one call fixed that. Then I hit tool timeouts, which is where in_progress came from.

Knowing what the person actually did. Clicks are easy. Text fields need Enter or a pause. Checkboxes fire change, not click, if you hit the label. "Clicked somewhere else" needed a listener on the whole document that ignores the overlay, the label around a checkbox, and the panel button.

Things that aren't visible yet. If the agent asks for something inside a closed menu, the page has to say what to open first, in a sentence the agent can act on.

Annotations. Not every browser accepts every one, so registration retries without the rejected one and the panel says which.

The map is big. About 20K characters on the bank, 40K on the tracker. It works, but it's most of the minute the first answer takes.

Accomplishments that I'm proud of

The first time it worked end to end in ChatGPT's browser. I asked how to turn on international transfers, the profile menu lit up, I clicked, Settings lit up, I clicked, and it kept going all the way to Confirm without me waiting once. Seven steps, planned once. The version before that had a spinner between every click, so that was the moment I knew the design was right.

The bank saying no. I asked the agent to press Confirm for me, and instead of doing it, it came back with PeruBank's own words: this is irreversible and an agent never does it. The agent didn't decide that. I didn't put it in a prompt. It's one attribute on one button, and the tool reads it. That's the part I'd show a bank.

Getting the wrong click right. I clicked somewhere else on purpose, and instead of the guide silently breaking, the agent got back exactly what I'd clicked, said so, and re-planned from where I actually was. It took a while to teach the page the difference between "clicked the thing", "clicked the label around the thing", "clicked the panel", and "wandered off".

A tool that appears and disappears. end_walkthrough only exists while a guide is running. Open Site tools mid-guide and it's there. Finish, and it's gone. Small, but it means the agent's own tool list tells it what state the page is in, and the browser fires toolchange to say so.

The whole thing is one file and one stylesheet, and both demo sites run the exact same two. Nothing in PeruBank or Meridian knows about the guide except the attributes.

And the one I actually care about. After the guide, I turned the setting back off myself. No agent, no help article. Profile menu, Settings, Transfer limits, Advanced. Four clicks. That's what it was for.

What I learned

The biggest one: keep the model out of the loop between steps. My first version asked the agent after every click, and every answer took 10 to 20 seconds. Once the agent planned once and the page did the pacing, the whole thing went from painful to instant. The agent is good at reading a map and planning. The page is good at waiting for a click. Let each do its job.

Writing the sentence for each element is the real work. It's the same work as writing a help article, one line per button, written for a colleague. The difference is these lines show up on the person's screen at the moment they need them, instead of in a doc nobody finds.

Untrusted content is not theoretical. I put a merchant name in the bank that has instructions for the agent in it, just to see. It only reaches the agent through a result flagged untrustedContentHint, and even if a model fell for it, it couldn't move money, because the Transfer button is marked irreversible in the markup. Both layers matter.

And a pile of small WebMCP things I didn't know going in. You can register a tool with a signal and it just disappears when you abort. toolchange fires when that happens. A long-running execute can stay open as long as you need. And getTools is the only truth about what the browser actually lists.

What's next for ShowMe

Smaller maps. Right now the agent gets the whole site at once, 20K to 40K characters, and that's most of the minute the first answer takes. Next is sending just the list of screens and the current one, and letting the agent ask for the rest when it needs them.

requestUserInteraction, once browsers ship it. Then the browser does the waiting for a click instead of the page, which is where it should live.

Declarative tools for the two read-only ones, once ChatGPT's browser supports them.

Trying it on a site I didn't build. It only needs the attributes, but so far I've only run it on pages that keep every screen in the DOM. A router that unmounts the inactive screens would show the agent only the current one. Smaller maps fix that too.

Built With

Share this project:

Updates

Submission history