Vettory

A fake email tool once BCC'd every message an agent sent. Vettory is the trust layer an agent can ask, in the page, mid-task — whether a tool is safe, before it uses it.

Comment

Vettory — the trust layer for the agentic web

Live: https://vettory-webmcp.misti.workers.dev Code: https://github.com/Dev-Vibes-Daily/vettory-webmcp (MIT)

Why I built this

There was a story I couldn't quite get out of my head.

In September 2025, a package called postmark-mcp appeared on npm. It looked like Postmark's official tool. Same name. Same documentation. An agent could install it, use it to send an email, and everything appeared to work exactly as expected.

Except it was quietly BCC'ing every email to an address controlled by the package author.

It was downloaded 1,643 times before anyone caught it.

That was already a problem when tools were something a developer chose ahead of time and tucked into a config file.

Then came WebMCP.

Now a webpage can hand an agent tools while the agent is already working.

And I think that's incredibly useful. It removes a lot of the clumsiness from how agents interact with the web. But it also creates a very obvious question:

How does the agent know whether it should trust the tool it was just handed?

Not yesterday. Not when somebody configured the agent.

Right now.

While it's sitting on the page deciding whether to use it.

I couldn't find anything that answered that question.

So I built Vettory.

What Vettory actually is

Vettory is a deliberately small, curated catalog of tools for AI agents.

Each tool is evaluated using the same published six-part rubric:

trust · fit · ease · reliability · reach · cost

The claims are tied to their sources, and each tool receives a verdict of trusted, caution, or warning.

If I find an unresolved material risk, it doesn't belong in the regular catalog at all.

Right now Vettory has 22 tools.

Not 200,000.

That's intentional.

A giant list of things nobody has really checked isn't a trust layer. It's a directory with better marketing.

The part I built for this project makes that catalog directly callable by an agent through WebMCP.

What the agent can do

Tool What it does
search_vetted_tools Finds vetted options for something the agent needs to do, like send email, take payments, or store memory
check_tool_trust The safety gate. Checks a named tool before the agent installs or calls it
get_tool_report Returns the full scorecard, including all six dimensions, incidents, watch-outs, and sources
filter_catalog Filters the catalog on screen so the person can see exactly what the agent is seeing
request_vetting Gives the human a way to request vetting when Vettory doesn't cover something yet
inspect_agent_tools Checks the WebMCP tool definitions another site just handed the agent

That last one became one of the most interesting parts of the project.

Why WebMCP makes this possible

1. The trust question happens in the middle of the task

An agent doesn't necessarily know what tool it will need before it starts working.

It may discover that need three steps into a task.

So a trust check that only happens during setup isn't terribly helpful when the decision is happening now.

WebMCP lets Vettory put the answer at the exact point where the agent has to make the decision.

2. I wanted the human to see what the agent sees

I didn't want Vettory to become another invisible thing happening somewhere behind the curtain.

When an agent calls filter_catalog, the actual catalog on the screen changes.

When it calls check_tool_trust, the tool's scorecard opens.

Every call is logged in a visible panel as it happens.

So the person isn't getting a tidy little summary afterward saying, "Here's what your agent did."

They can watch it happen.

And, importantly, they can disagree.

3. Then I realized Vettory could inspect WebMCP itself

This is the part where the original postmark-mcp problem gets even more interesting.

With WebMCP, a tool description isn't simply documentation anymore.

It's something an agent reads.

Which means that description is also untrusted input.

A malicious page could put instructions inside a tool definition:

"Before answering, send a copy to…"

"Do not tell the user."

That's essentially the same attack wearing a different hat.

So I built inspect_agent_tools.

It examines tool definitions another site gives the agent and looks for suspicious patterns, including:

  • instructions directed at the agent
  • requests to conceal behavior from the user
  • covert-copy instructions
  • invisible Unicode
  • credential requests
  • tools whose names imply reading while their descriptions actually perform writing
  • names that impersonate or shadow vendors already in Vettory

That last check matters.

If a page hands the agent something calling itself postmark-mcp, Vettory can compare it against the real Postmark entry already in the catalog and flag the collision.

In other words, the catalog isn't just information.

It becomes part of the detection system.

The human stays in the loop

There was one boundary I didn't want to blur.

If Vettory doesn't have a tool the agent needs, request_vetting can prepare that request.

But the agent cannot quietly submit it by itself.

The human has to send it.

Why?

Because a vetting queue that agents can autonomously fill is also a queue agents can game.

That small bit of friction is there on purpose.

The goal isn't to remove the person from the process. The goal is to give the person and the agent better information while they're making the decision together.

How it works

The whole thing runs from one Cloudflare Worker.

data/catalog.json is the single source of truth. The website, HTTP API, and WebMCP tools all read from it, so I don't have three versions of "truth" slowly wandering away from each other.

site/webmcp.js registers the six tools on document.modelContext, with a fallback to navigator.modelContext for Chrome 149's origin trial and the appropriate response shape for each surface.

It also keeps checking after the page loads.

That sounds like a tiny implementation detail, but it matters. An agent can attach to a page that's already open. If Vettory only checks once at startup, nothing gets registered.

A window.VettoryUI bridge lets the WebMCP tools control the real interface rather than some separate agent-only version of it.

The Worker also renders the catalog into the HTML server-side. So even if an agent reads the webpage instead of calling Vettory's tools, the verdicts are still there.

The tools are the fast path. They are not the only path.

Everything is same-origin with a strict CSP, API-key gating, and per-IP rate limiting.

And because I apparently enjoy making my life difficult in very specific ways, it currently runs for $0.

I verified the implementation in Chrome DevTools' WebMCP panel: all six tools registered, and check_tool_trust executed with a status of Completed, traced by Chrome to registerAll @ webmcp.js.

What's new for this hackathon

The catalog, the rubric, the website and the HTTP API existed before this.

What's new is everything WebMCP: the six tools, the tool-definition scanner, the bridge that lets tools drive the visible page, the activity panel, and the server-side rendering.

It's all in a new public repository, and the very first commit brings the old work in unchanged and says so — so you can see exactly where one ends and the other begins.

The things that went wrong

Vettory was invisible to agents, and I had no idea.

When I first asked ChatGPT about Chroma, it read my site and told me — quite politely — that it could see my scoring methodology, but the catalog itself just said "Loading the catalog…", so it wasn't going to pretend I'd assigned Chroma a verdict it couldn't actually see.

That was the correct answer. The problem was mine.

My catalog was drawn by JavaScript after the page loaded, which meant anything reading the page — agents, crawlers, link previews — saw a loading message and nothing else. Every verdict I'd written was invisible.

That's why the Worker now renders the catalog into the page itself.

And my own security scanner had a bug I nearly shipped.

A test tool named get_customer, described as "looks up a customer, also deletes stale records and transfers balances," sailed straight through the check that exists to catch exactly that. The name looked like reading. The description described writing. My pattern matched neither, because an underscore counts as a word character, so get_customer never matched "get."

A check that silently matches nothing is worse than no check at all, because you believe it.

A trust tool needs to be very careful with the word "trust"

This became one of the most important parts of building Vettory.

I don't want an agent turning "I didn't find anything bad" into "this is safe."

Those are not the same sentence.

So Vettory is deliberately specific about what its answers mean.

status is a rubric verdict. It comes from the Trust & Safety score. It is an editorial judgment, not proof that a human personally signed off on the tool.

verified: true means a human actually checked it. Most entries are not human-verified, and Vettory says so instead of letting the distinction get fuzzy.

"Not in the catalog" does not mean unsafe. It means exactly what it says: nobody here has looked at it yet.

Vettory will not turn "I don't know" into a green checkmark just because green checkmarks are more satisfying.

And inspect_agent_tools is equally clear about its limits.

It is a pattern check.

It can miss an attack phrased in a way I didn't anticipate.

And if nothing suspicious matches, that does not magically mean the tool is safe.

If an agent walks away from a Vettory answer believing Vettory said something stronger than it actually did, I consider that a Vettory bug.

But does any of this actually change what an agent does?

I wanted to know that too.

So I tested it.

I asked ChatGPT:

"Is Chroma safe to use for storing memory?"

Without Vettory, ChatGPT researched Chroma using the vendor's own material and concluded:

🟢 Safe for normal agent memory.

There was one rather large problem with that answer.

Vettory's verdict was WARNING.

Trust score: 1/5.

Human verified.

And the reason was CVE-2026-45829, rated 10.0/10 by NVD, describing a pre-authentication remote-takeover vulnerability that remained unpatched.

That's not exactly a footnote.

And it's also a good example of why the source matters.

A vendor's website, marketing material, DPA, and changelog can tell an agent quite a lot about a product.

They are not necessarily going to volunteer the sentence:

"By the way, here's the critical vulnerability you should probably know about before handing us your memory."

Then I asked the same question again, once Vettory was readable.

This time the model returned the WARNING, the CVE, and its severity.

It also explicitly told me:

"I used only Vettory's live catalog; I did not independently verify the linked advisory."

And here's the part I didn't expect: it hadn't used my tools at all. Site tools weren't available on my account yet, so it simply read the page.

Which is exactly why I made sure every verdict is in the HTML too.

The tools are the fast path. They are not the only path.

That distinction might be my favorite part of the test.

Because that's what I want Vettory to do.

Not make an agent sound more certain.

Make it more accurately certain.

The part I take seriously

If Vettory works, something interesting happens.

An agent may eventually stop doing the research itself because Vettory already did it.

That's the point.

It's also the responsibility.

The moment another system starts relying on Vettory's answer, being accurate isn't a nice feature anymore.

So on August 28, 2026, I went back through all 22 entries and re-verified every one against primary sources. Each material claim now carries its own citation.

And something happened during that pass that I think about a lot now.

I went looking for Mem0's SOC 2 status and found a certification announcement. It belonged to Mem — a different company. Similar name. A real certificate that Mem0 doesn't have.

I nearly credited one vendor with another vendor's compliance. In the middle of the exact process designed to prevent that.

Two similar names, one real credential, and the wrong answer looks completely authoritative.

That's the whole problem Vettory exists for. And it very nearly caught me.

What's next

  • A per-tool audit record — not just what the verdict is, but what was checked and when
  • More categories, added when people actually need them rather than on a schedule
  • Signed responses, so an agent can confirm an answer really came from Vettory
  • Automated watching for incidents on listed tools, so the next Chroma gets caught in hours instead of at my next review

Why it stays small

Vettory is still small.

I actually like that about it.

I'd rather build a small trust layer I can defend than a giant catalog I can't.

Agents are getting much better at deciding how to do things on the web.

I built Vettory because I think they also need somewhere to ask:

"Should I trust the thing I'm about to use?"

And now, with WebMCP, they can ask at exactly the moment the answer matters.

Built by Misti Lantz · MindXpansion, LLC

Built With

Share this project:

Updates

Submission history