Inspiration

Research and diligence agents are only as good as their searches. When a search comes back without the answer, the agent doesn't know it. It writes the wrong number into the memo with total confidence. The industry's answer has been to ask more engines at once and hope the answer is somewhere in the pile. That's expensive and slow, and it still never confirms the answer is actually there.

Telem’s fetch already catches blocked and empty pages and retries them on another fetcher. Search had no equivalent.

What it does

Backstop is a verified-search layer built on Telem. Instead of fanning out to every engine, it asks one, verifies the result, and only spends more when it needs to.

  1. Routes: sends the first search to the engine our data shows is strongest for that type of question (Perplexity for factual questions, Exa for finance).
  2. Verifies: a fast two-stage checker confirms the answer is really in the results. A lightweight decision model handles most checks cheaply, and an LLM checker takes over when it's unsure.
  3. Stops: the moment the answer is confirmed, the agent gets the evidence and moves on.
  4. Rescues: on a miss, it applies the fix that matches the failure. It retries on the next engine in our measured rescue order with a rewritten query, adds a date filter when results are stale, or reads the page with Tavily Extract when the snippet is too thin.
  5. Refuses to guess: if nothing surfaces the answer, it returns "not found" instead of letting the agent make something up.

Adoption is one line: swap AsyncTelem() for BackstopTelem(). Backstop also ships a hosted API with keys, metering and rate limits, and our site races it head to head against Telem's default routing on any question you type.

On 200 held-out questions, Backstop matched Telem's default routing on accuracy (94.5%) at 53% lower cost per question, with 84% fewer engine calls and a 15% faster median answer. It even cost less than a single-engine agent with no checking at all.

  • Accuracy: Backstop 94.5%, Telem default 94.5%, single engine 94.0%
  • Cost per question: Backstop $0.048, Telem default $0.102, single engine $0.063
  • Engine calls per question: Backstop 1.31, Telem default 8.00, single engine 1.86
  • Median time per question: Backstop 7.3 s, Telem default 8.6 s, single engine 9.0 s
  • Search results the agent had to read: Backstop 1,521, Telem default 8,704, single engine 2,105

How we built it

Backstop is Telem-native from end to end:

  • One API, six engines: Tavily, Exa, Brave, Perplexity, Parallel and Ceramic, all pinned through Telem. That's how we built a 330-question by 6-engine hit matrix in minutes, and it became our routing and rescue data.
  • Telem's own accuracy routing as the baseline we race against.
  • Goals, sessions, context and metadata on every call, so every Backstop run, including each rescue step and verdict, is labelled and grouped in the Telem console.
  • Telem fetch as the page-read fallback.

Tavily powers two layers: it's one of the six engines, and Tavily Extract pulls the exact supporting passage from a page so the checker can verify against real evidence.

The checker cascade combines a fast decision model with Claude Haiku, and the agent runs on Claude Sonnet. The backend is Python, asyncio, pydantic and FastAPI. The live race streams both agents side by side over server-sent events, and the site is plain HTML, CSS and JavaScript on Vercel. Every search writes a replayable event log, and the codebase has over 800 automated tests.

We built the evaluation to hold up to scrutiny:

  • Held-out numbers only: every routing and rescue decision was learned on a separate 100-question split, and every number we report comes from 200 held-out questions plus 30 false-premise traps.
  • Clean results: benchmark-copy pages were filtered out of every setup's results before anything saw them.
  • Exact-list scoring: the checker is graded on the exact result lists it saw.
  • Apples to apples: the same agent model, prompt and tools in every setup. Only the search step differs.

Challenges we ran into

  • Benchmarks leak onto the web. Public answer sheets for SimpleQA and FinanceBench show up in live search results. We built a contamination filter that strips them from every setup, so every number reflects real search.
  • Tuning model output budgets. Built-in reasoning can quietly consume a model's output budget. We traced it through cached responses and tuned our calls so verdicts and grades come back compact and reliable.
  • Agents at evaluation scale. Running 920 agent runs meant handling messy model output, making every run resumable, and making sure the test suite could never touch paid APIs.
  • Real-time races on serverless. Streaming two live agents side by side on Vercel meant keeping each race inside a single request, with hard cost and time caps.

Accomplishments that we're proud of

  • Same accuracy at half the cost, measured on questions we never tuned on.
  • A full evaluation: 920 agent runs, with the cost of every model call and engine call counted, checker included.
  • A checker that catches three out of four genuine misses while raising false alarms on under 3% of good results, at about $0.0005 per question.
  • On 30 false-premise trap questions, Backstop flagged the false premise every time.
  • A real product, not a prototype: a one-line drop-in client, a hosted API, a live race anyone can try, and over 800 automated tests.

What we learned

  • Misses travel together. When one engine misses, others often miss too, which is why fanning out to everyone is such expensive insurance. A targeted check plus a targeted rescue gets the same accuracy for far less.
  • Verification pays for itself. A check costs about $0.0005 per question, and stopping as soon as an answer is confirmed is what cuts the bill in half.
  • The real cost of search is what the agent has to read. Telem's default handed our agent 8,704 search results across the test set, and Backstop handed it 1,521. Fewer, better results mean faster, cheaper answers.

What's next for Backstop

  • A rescue table that learns from every search. Each check records which engine missed and which one caught it. That data compounds with every search, and no single engine or pre-result router can collect it.
  • Verified search as a routing mode in Telem: check first, expand only when needed.
  • Coverage beyond factual questions, into open-ended research and news.
  • Usage-based pricing per verified search for research, finance and legal teams, with an exportable audit trail behind every answer.

Built With

  • anthropic
  • asyncio
  • brave-search
  • ceramic
  • claude
  • css3
  • exa
  • fastapi
  • html5
  • httpx
  • javascript
  • matplotlib
  • openai
  • parallel
  • perplexity
  • pydantic
  • pytest
  • python
  • server-sent-events
  • sqlite
  • tavily
  • telem
  • uvicorn
  • vercel
Share this project:

Updates

Submission history