Shared Reference Canvas

Inspiration

I have had a really hard time describing a UI to an agent. It understand teh world in a different way than i do, i usually point to things, mention 'here, there, this thing'. To an agent that is meaningless. I wanted to figure out a way to have a shared language that makes agent collaboration as seamless as possible, and i focused on working in a canvas for an agent.

What it does

When you select anything on the canvas it creates a short lived keyword that becomes the reference between you and the agent for that specific selection. You can select a group of elements and it creates a shared reference for the group, you can also select different empty regions. This makes describing things to an agent really simple when you are working with it, and you dont have to be sending out screenshots of the Ui for the agent to understand what you are talking about.

How we built it

I used NextJS, WebMCP, and Codex to build the infinite canvas with different media types. The keywords are meant to be stored locally and all the mcp tools as well so this is a really lightweight implementation that does not require having a stored state, or stored realtime session in a db. There are also dynamic tools when you double click on any element depending on its type, so you can have the agent operate differently depending on media type.

Challenges we ran into

I would say the biggest challenge was making the tools description and names as relevant as possible for the agent to make sense of them in a way that felt natural to interact with. In the beginning it would find the keywords but not have the necessary context on how the app worked, so its interactions with the user where sometimes confusing. In the beginning i solved this by giving the agent a tool with a really long description that explained the app, the tool was not meant to be used, it was jsut to have the agent read the context of the app.

Accomplishments that we're proud of

I am really proud of how natural the interaction with the agent felt at the end. When i began playing around with the canvas and the agent, it felt really simple to just instruct the agent on what i wanted without having to overcomplicate my speech trying to describe an element.

What we learned

I learned how WebMCP works, this was my first project for the challenge. Having the mcp be registered in the html make sit so that any change in the UI can be immediately referred to by the agent. I also learnt about the Abort Signals for the agent, for destructive actions.

What's next for Shared Reference Canvas

I would love to see this become some sort of package where maybe its a extension for the browser or an app where any interface we use can also have these shared references as we everyday talk more with agents, and have to spend a painful amount of time describing our world that can sometimes be confusing to describe.

Built With

• codex
• nextjs
• vercel
• webmcp