Agent Lost & Found

A WebMCP-powered lost-and-found where agents investigate clues, compare evidence, and narrow the search — while humans make the final decision.

Comment

Inspiration

When I was a kid, I loved treasure maps.

Someone would hide something and leave you a clue. You would search, realize your first guess wasn't quite right, discover another clue, and slowly narrow down the possibilities.

Looking back, I think the part I enjoyed most wasn't actually finding the treasure.

It was getting closer.

When I started experimenting with WebMCP, that memory came back to me. I wanted to find a use case where an agent wasn't simply pressing buttons faster than a human, but was actually helping with the reasoning between those actions.

Lost and found felt like a surprisingly natural fit.

When we lose something, we rarely remember perfect database fields. We remember fragments.

“I lost something on a key ring.”

Then, after thinking about it:

“Wait, it had a small bear charm.”

That became Agent Lost & Found: an experiment in what happens when a human and a browser agent investigate the same problem together.

What I built

Agent Lost & Found is a lost-and-found catalog with WebMCP built directly into the website.

People can still browse the catalog like a normal website.

But a WebMCP-enabled browser agent doesn't have to interpret search boxes, filters, buttons, and item cards to understand what the site can do.

Instead, the website exposes seven structured investigation tools:

search_lost_items
get_item_details
get_search_facets
compare_items
get_match_evidence
get_active_case
request_claim

Together, these let an agent search the catalog, inspect candidates, compare possible matches, analyze evidence, find useful follow-up clues, resume an existing investigation, and eventually ask the human to review a possible match.

--

From a vague memory to a specific match

The demo begins with an intentionally incomplete clue:

“I lost something on a key ring.”

There isn't enough information to immediately know which item is correct.

And I didn't want the agent to pretend otherwise.

Instead, it can search the catalog, inspect the remaining candidates, and use the available information to narrow the investigation.

Then the human remembers one small detail:

“It had a small bear charm.”

That clue doesn't start a new search.

It becomes part of the existing case.

The candidates are evaluated again, and LF-017, a bear keychain, moves toward the top of the investigation because the new clue provides much stronger evidence for it.

This simple interaction became the core of the project for me.

The agent doesn't need the human to remember everything at once.

The human doesn't need to manually restart the search every time another detail comes back.

They build the investigation together.

Making the ranking explain itself

Once I had progressive clues working, another problem appeared.

If LF-017 moves up after I mention the bear charm, the interface shouldn't just show me a different ranking.

It should be able to explain what changed.

So each investigation step stores score snapshots and evidence breakdowns.

When the evidence changes, Agent Lost & Found can compare the new state with the previous one and show Candidate Movement.

A candidate may move up because a new clue matches.

Another may move down because the new information contradicts it.

Evidence is separated into three states:

  • Matched
  • Unknown
  • Contradiction

That distinction became more important than I expected.

If the catalog doesn't contain information about a particular feature, that doesn't automatically mean the item contradicts the human's memory.

Sometimes the correct answer is simply:

We don't know yet.

For the same reason, the number shown by the application is called a Match Score.

It is a deterministic matching score based on the available evidence — not an AI confidence probability.

I wanted the user to be able to inspect why an item ranked highly instead of being asked to trust an unexplained number.

The Casefile

Once the investigation became multi-step, I ran into another very ordinary problem:

I refreshed the page.

Everything disappeared.

That didn't feel right.

If a human and an agent are investigating something together, the investigation itself should exist independently of a single page session.

So I introduced a versioned Casefile.

It stores the active investigation, including the original description, clues, candidate state, score history, ranking history, timeline, strongest match, and claim-review state.

After a refresh, the application validates and restores the Casefile.

The agent can then call:

get_active_case

and continue from the same investigation.

One small detail mattered to me here:

restoring state should not mean inventing history.

The application restores the Casefile, but it does not replay fake tool calls or create Agent Activity for actions that never happened.

The investigation continues from real state.

Why WebMCP?

I understood WebMCP differently after building this project.

At the beginning, I mostly thought:

“Websites can expose actions as tools.”

That's useful, but it turned out not to be the most interesting part.

For me, the more interesting idea became shared state between the website, the human, and the agent.

Without WebMCP, an agent interacting with a lost-and-found website would need to interpret an interface built for humans: locate the search field, understand the filters, inspect cards, and infer what changed after each interaction.

With WebMCP, the website can expose its actual capabilities as structured tools while keeping the normal visual interface for people.

More importantly, both operate on the same underlying investigation.

The human interface and WebMCP tools share the same search, matching, evidence, ranking, and Casefile logic.

When the agent investigates something, the human can see the result.

When the human adds or corrects a clue, the next agent action sees that updated state.

That was the point where the project stopped feeling like:

“AI controlling a website.”

It started feeling more like:

a human and an agent using the same product together.

The tool I deliberately didn't build

There was one tool I deliberately left out.

If the agent finds a strong match — for example, LF-017 after the bear-charm clue — should it be allowed to claim the item automatically?

Technically, I could expose:

confirm_claim

I decided not to.

The agent can call:

request_claim

which opens the claim review interface for the human.

But the final confirmation still requires a human action.

This isn't a technical limitation.

It's a product decision.

Searching through candidates is useful work to give an agent.

Comparing evidence is useful.

Finding the next clue worth asking about is useful.

But deciding that a real item belongs to someone has consequences.

Building this made me realize that not every action that can become an agent tool necessarily should become one.

That eventually became the principle behind the project:

Agents investigate. Humans decide.

How I built it

Agent Lost & Found is built with React, TypeScript, Vite, and WebMCP, and deployed on Cloudflare Pages.

For this hackathon version, the application uses a deterministic catalog of 30 lost items backed by structured metadata.

The catalog is intentionally small.

I wanted the investigation to be reproducible and inspectable. A judge should be able to understand why LF-017 appeared rather than having the result hidden behind a huge dataset or an opaque model.

The human interface and WebMCP layer share the same underlying engines for:

  • search and normalization
  • matching and ranking
  • evidence classification
  • clue handling
  • score snapshots
  • candidate movement
  • Casefile persistence

I also built a deterministic demo/presenter mode for evaluation.

It fixes the demo environment so the same scenario can be reproduced reliably.

It does not preselect the user's query, winning item, Match Score, or final claim decision.

Those still come from the investigation at runtime.

Challenges I faced

One of the biggest surprises was that getting a WebMCP tool to return an item wasn't particularly difficult.

Getting seven tools to behave like parts of the same investigation was much harder.

I kept running into small questions that turned into important product decisions:

  • What happens when the human remembers another clue?
  • What if the human corrects something from earlier?
  • Is missing metadata a contradiction, or just unknown?

I also had an earlier version where the demo scenario worked much better than unrelated searches.

That bothered me.

I didn't want to submit a project that only worked when I typed exactly the right sentence from the demo video.

So I moved the search toward a generic, metadata-driven system instead of special-casing individual item IDs.

The deterministic demo environment stayed, but the investigation itself still had to work through the same underlying engine.

That made the project less magical internally, but much more honest.

What I learned

The biggest thing I learned is that building an agent-friendly website isn't just about exposing more tools.

Adding tools might actually be the easier part.

The harder questions are about coordination and boundaries:

What does the agent know?

What does the human know?

What state do they share?

Before this project, I mostly thought about agents in terms of what they could automate.

After building it, I'm much more interested in interfaces where agents handle repetitive investigation while humans contribute memory, context, correction, judgment, and final control.

The most useful moment in the demo isn't really when LF-017 appears.

It's the moment between:

“something on a key ring”

and:

“a small bear charm.”

The human didn't have enough information at first.

The agent didn't have enough information either.

But together, they got closer.

That's the part of WebMCP I wanted this project to explore.

What's next

Agent Lost & Found is intentionally a focused prototype.

The current 30-item catalog makes the investigation easy to reproduce and inspect, but a real lost-and-found system could contain hundreds or thousands of visually similar objects.

That's where this interaction model could become much more useful.

I'd like to explore larger catalogs and eventually image-derived clues, where a photo could become another piece of evidence in the Casefile.

But I would keep the same boundary I ended up with here.

The agent doesn't need to make every decision.

Sometimes the better product is one where it helps you investigate the possibilities, understand the evidence, and get close enough to make the final decision

Agents investigate. Humans decide.

Built With

Share this project:

Updates

Submission history