Understudy

• 

Inspiration

My wife is a dentist. A part of her week has nothing to do with teeth. She works with the SEO people on the clinic's articles, going down a long list and tagging each one by what is trending, or sorting them into the right category.

The tagging is easy. Knowing which topics are trending this month is not, and that knowledge is not in the tool she is clicking in. It is in a browser tab next to it.

So she works in two windows. Search in one, tag in the other, back and forth, one row at a time. The switching costs more than the actual work does. Next month there is a new list and she does it all again.

She cannot automate it. Not because the routine is complicated, she knows it better than anyone, but because automating anything means writing code, and that is not her job. So it stays manual forever. She asked me once why the computer could not just do it. I did not have a good answer.

So there are three things here and none of them sit in the same place. A model already knows what is trending. The page already knows how to tag. And the routine itself is not written down anywhere, it is just in her head. I wanted to find out if the page could pick it up by watching her do it one time.

What it does

Understudy is a triage workspace for 1000 Hacker News stories. It has six WebMCP tools when it loads. It can have more by the time you are done with it, because you make them.

You press Teach and do a routine by hand. Filter for "amazon", 14 items match, tag them "big-tech", set them to high priority, done. The page records that as four commands in its own vocabulary, not as clicks or screen positions.

Then you tell the agent to turn it into a skill. It reads the recording and proposes a tool called triage_title_matches with three parameters: title_phrase, tag, and priority. You look at the schema and correct it. I wanted only the phrase to change between runs, so I unbound tag and priority with two clicks and approved it.

Now the page calls registerTool and that tool exists, in a live session, about ninety seconds after it did not.

The last step is the one worth watching. I asked for "every other big tech company". I never said which ones. The agent named Google, Apple, Meta and Microsoft on its own and ran my tool once per company. I taught it on 14 items. It handled 115 more, using knowledge the page has no way to hold.

How I built it

Plain ES modules. No build step, no framework, no dependencies. You can serve it with python3 -m http.server.

Everything in the interface goes through one command bus. A click and an agent call both end up dispatching the same command, so there is exactly one code path to record and one to replay.

There are two layers of WebMCP:

A fixed set of six tools registered when the page loads: start_recording, stop_recording, get_recording, save_skill, list_skills, run_skill. Read only ones are marked with annotations.readOnlyHint.

One real tool per approved skill, registered at runtime with document.modelContext.registerTool. Each one is held by an AbortController, so revoking a skill actually unregisters its tool rather than leaving a dead entry behind.

The most important rule in the whole project is that the agent never writes the steps. save_skill takes a recording id and a list of bindings, saying which field of which step becomes which parameter. The page builds the JSON Schema from that. The steps are what I did, unchanged. A learned skill therefore cannot do anything I did not demonstrate. That is the safety story and the reliability story at the same time, and it comes from the structure rather than from a check.

Challenges I ran into

The biggest one took a day to understand. A tool registered in the middle of a turn is callable, but the first call fails. ChatGPT's browser works from a list of tools it captured when the turn started, and the new tool is not in it. The agent explained it better than I could: "The first immediate call failed because the existing tool snapshot was stale; after refreshing the registry within the same turn, the call succeeded." That recovery cost about fourteen seconds.

I lost most of a day to it because I was certain the bug was mine. I rewrote the schema. Then the tool name. Then I added waits after registerTool, which did nothing, because waiting was never the problem. The tool was registered fine. The agent was just holding an older list.

So learned skills go through run_skill, which is registered at page load and is always in the snapshot. A skill you just taught is callable right away.

Two smaller ones. The toolchange event fires in Chrome 149 and later but does not exist in ChatGPT's browser, where modelContext is not an EventTarget, so nothing that depends on it can be relied on. And prompt, confirm and alert do nothing at all in the in-app browser. They fail silently, with no error, which is a miserable thing to debug. Every question the app needs to ask is asked inline in the page instead.

I wrote a small probe page, spike.html, to answer these properly instead of guessing. It is in the repo.

Accomplishments that I'm proud of

The moment where the agent names four companies I never mentioned and runs a tool I taught it two minutes earlier. Neither side could do that alone. The page knows how to filter and tag but has no idea what a big tech company is. The agent knows the category but has no reliable way to drive an interface.

I am also glad the human stays in charge of the schema. The agent proposes parameters, and you change them before the tool exists. It suggested three. I wanted one.

What I learned

Recording clicks does not work. Recording intent does. Because every command in the app acts on the current matching set rather than on specific item ids, a recording generalises for free. SET_FILTER {contains:"amazon"} has an obvious parameter in it. SELECT_ITEMS {ids:[3,7,12]} never will. That one decision is what makes learning from a single demonstration possible at all, and it also happens to be a better way to build bulk triage.

I also learned to test against both clients early. Two of the three problems above would have been invisible in Chrome alone.

What's next for Understudy

Skills are stored per browser today. Being able to share one, or bring your own set to a page, is the obvious next thing.

Recordings only capture set based commands, which is deliberate, but nested parameters and a way to chain one skill into another would cover more real routines.

And there are three things I marked as open rather than claimed: whether AbortController revocation reaches the agent, whether readOnlyHint changes anything observable, and where the tool count ceiling is. Those need a longer test than a hackathon week allows.

Built With

• chrome
• css3
• html5
• javascript
• localstorage
• openai
• webmcp