VibeCheck: what people actually said

Inspiration

This came out of a habit I didn't think about much until I tried to explain it to someone.

When I'm actually deciding on something — a laptop, a bank, a framework — I don't trust the first page of results. I search for it, then I add "reddit" to the end and go read threads until I've got a real picture in my head. Marketing pages tell you what a product is supposed to be. Threads tell you what it's actually like to own the thing.

I wasn't trying to build another search tool or another recommendation engine. There's no shortage of those and most of them aren't much help when you're actually trying to decide something. What I wanted to build was the reading part — go gather what real people are saying, put it somewhere useful, and then get out of the way. Because there's one part of this I don't think a model should be doing for you, and that's deciding who you trust.

WebMCP came into the picture later than the idea itself, but once it clicked, it kind of reframed the whole thing. Say you're researching a laptop and you've got an agent helping. Halfway through, you notice one of the sources reads like an affiliate blog wearing a Reddit post's clothes, so you reject it. Or you find one comment that nails the exact problem you're worried about, so you pin it. None of that exists anywhere except in your head and on your screen, right then. It's not sitting in a database somewhere, and it's definitely not something you'd think to type into a prompt.

WebMCP is what lets the agent reach into that. Not a summary of it, not a description of it after the fact — the actual state, live, while you're both looking at the same board.

That's also, concretely, what it buys you that you didn't have before. Reject a source mid-conversation and the agent doesn't need you to explain what changed — its next call against the board already sees the claim re-weighted, because it's reading the same object you just edited. And it runs the other way too: the agent can pin a claim or apply a constraint on its own, and instead of that decision getting buried three messages back in a chat log, it shows up as a card on your board, right where you'd notice it, and you can undo it with a click if it got it wrong. Neither of you is re-explaining state to the other. You're both just looking at the same thing.

That turned into more than an integration detail for me. It's a big part of why the whole app is shaped the way it is.

What I learned

Honestly the biggest lesson wasn't technical. It's that "gather opinions" is a lot harder than "run a search."

A search API hands you results. Turning those into something usable is where it gets messy. Ask it "is the Mac Mini a good device for developers?" and you'll get close to nothing back. Strip it down to "mac mini developers" and suddenly there are real threads with real replies. Search engines want keywords, not questions, and getting from one to the other without losing what the person actually meant took more tuning than I expected going in.

Then there's the discussion-vs-not-discussion problem. A big chunk of what came back technically matched the query but wasn't a discussion at all — Facebook groups, LinkedIn posts, promotional junk dressed up to look like a real thread. I ended up writing a whole layer just to translate a person's actual sentence into something the search API would respect, and then another pass to figure out which of the results were even worth reading.

Trust modeling was the other one. My first pass at "how strong is this claim" was just a source count — more sources, stronger claim. Which sounds fine until you notice that three people copying the same affiliate article isn't three opinions, it's one opinion wearing a disguise three times.

So I moved to weighting the evidence instead of counting it:

$$ \text{raw}(claim) = \sum_{s \in \text{evidence}} w(s) $$

where

$$ w(s) = \begin{cases} 0 & \text{you rejected it} \ 1.5 & \text{you trusted it} \ 0.4 & \text{it's a low-signal source} \ 1 & \text{ordinary discussion source} \end{cases} $$

and then normalized against whatever the strongest claim on the board is:

$$ \text{weight}(claim) = \min\left(1,\ \frac{\text{raw}(claim)}{\max_{c \in \text{claims}} \text{raw}(c)}\right) $$

Once that was in place the UI more or less designed itself. The support bar isn't a decoration sitting on top of the claim — it's that formula, visible. Reject a source and the segment it owned just goes away, and you can watch a claim that looked like consensus a second ago drop down to "thin." That's probably my favorite thing in the whole app. It makes "trust" a thing you can actually see move, instead of a number some model handed you with no way to check its work.

How I built it

I kept the architecture small on purpose. One state shape, one reducer — and both the UI and the six WebMCP tools go through that same reducer to touch the board. Nothing gets a side door.

That constraint came from something specific I wanted to avoid: a tool that quietly changes something you can't see. The second that happens, this stops being collaboration and starts being something closer to surveillance, and I didn't want to build that even by accident.

The actual pipeline under the UI is smaller than you'd guess. SerpApi's google_forums engine goes and finds the threads first. Then an LLM — routed through OpenRouter, free model first with a paid one as backup — does three things with what comes back: decides whether a thread is actually relevant, figures out whether the person sounds like they've used the thing versus just repeating something they heard, and clusters the recurring points into claims with the evidence attached to each one. Everything downstream of that — the reducer, the support math, the whole board — is only as good as that step, so it ended up eating far more of my time than I planned for.

Challenges

SerpApi's free tier is 250 searches a month, so credit discipline wasn't optional. I built it as two stages — google_forums runs alone first, and only if that comes back too thin does a second, broader search fire. There's also a disk cache for local dev, so I'm not burning real credits every time I tweak a font size. Deciding what "too thin" even means turned into its own small project, since real google_forums results are noisier than I expected — Facebook and LinkedIn show up constantly, looking like discussion when they're really not, which pushes the fallback to trigger more than I originally planned for.

The more frustrating problem was getting the model to stop finishing other people's sentences. Search snippets get cut off mid-thought, sometimes literally ending in "…", and a language model's instinct is to complete that thought for you. Normally that's a fine instinct. Here it's dangerous — if a source says "the battery life is good, but…" and the model decides the rest was probably "not great under heavy load," that's a completely plausible sentence sitting right next to a citation, and it's also something nobody ever actually said. I had to write out very literal rules against it: don't finish a cut-off snippet, don't invent a detail that isn't in the text, don't borrow a phrase from one source's snippet into another source's line. None of that is clever prompting. It's just closing off, one by one, all the small ways a model fills gaps with confidence instead of silence.

The last challenge is the one with no clear finish line, which is making the board itself worth spending time on. It's not hard to build something that's technically correct and still nobody wants to open twice — an accurate list of claims with checkboxes next to them is not, on its own, a reason to come back. The support bar had to feel like it was actually moving when you rejected something, not just updating a number. Rejecting a source needed to feel like a decision, not a form field. Even the empty state needed a reason for someone to click something instead of staring at a blank input box. None of that had a spec. It was mostly just looking at it, deciding it wasn't quite right, and doing it again — and if I'm honest, that's the part I kept circling back to more than anything else in the project.

Built With

• chatgpt
• claude
• css3
• model-context-protocol
• node.js
• openrouter
• react
• serpapi
• typescript
• vercel
• vite
• webmcp
• zod