VibeCheck: what people actually said

Inspiration

It started from a habit I didn't think about until I tried to explain it: whenever I'm deciding on something real — a laptop, a bank, a framework — I don't trust the first page of search results. I type the question, then I add "reddit" to the end of it, and I read threads until a picture forms in my head. Marketing pages tell you what a product wants to be. Threads tell you what it actually is.

That habit is the whole product. I wasn't trying to build a search engine or a recommendation engine — those already exist and mostly disappoint. I was trying to build the reading part: gather what real people said, and then get out of the way while a person does the one thing a model can't do for them, which is decide who to believe.

The WebMCP angle came second, but it clicked hard once I saw it: an agent helping you decide on a laptop is useless if it can't see that you already rejected one source as a shill, or that you pinned a claim as the one that actually matters to you. That state lives in your head and on your screen — not in a database, not in a prompt. WebMCP is the only way an agent gets to touch it directly, in your session, live. That's not a nice-to-have integration; it's the reason the product exists in this shape at all.

What I learned

The biggest lesson wasn't technical — it was that "gather opinions" is a much harder problem than "run a search." A raw discussion-source query returns a pile of ambiguity: half the results are Facebook groups and LinkedIn posts dressed up as discussion, snippets are truncated mid-sentence, and search engines match your question almost literally, so a natural question like "is the Mac Mini a good device for developers?" returns nothing while "mac mini developers" returns real threads. I ended up writing a whole normalization layer just to translate a person's actual sentence into something a search API will respect.

The second lesson was about trust modeling. Early on, "support" for a claim was just a count of sources. That's wrong — three people reposting the same affiliate blurb isn't consensus, it's an echo. So the real signal isn't $n$, it's a weighted sum:

$$ \text{raw}(claim) = \sum_{s \in \text{evidence}} w(s), \qquad w(s) = \begin{cases} 0 & \text{rejected by you} \ 1.5 & \text{trusted by you} \ 0.4 & \text{low-signal source} \ 1 & \text{ordinary discussion source} \end{cases} $$

and the bar you see is that raw score normalized against the best-supported claim on the whole board:

$$ \text{weight}(claim) = \min\left(1,\ \frac{\text{raw}(claim)}{\max_{c \in \text{claims}} \text{raw}(c)}\right) $$

The moment I had that formula, the UI basically designed itself: the support bar isn't decoration, it's the math made visible, and rejecting a source visibly collapses the segment it owned. Watching a claim go from "consensus" to "thin" in real time, just because you clicked reject on one Facebook post, is the whole pitch of the app in one interaction.

How I built it

The architecture is deliberately small: one state shape, one reducer, and both the UI and the six WebMCP tools read and write through it. That constraint — "nothing the agent does is invisible on screen, and nothing you do on screen is invisible to the agent" — came from a very specific failure mode I wanted to avoid: tools that quietly mutate state the human can't see, which turns "collaboration" into "surveillance."

The pipeline underneath is two calls: SerpApi's google_forums engine finds the threads, and an LLM (routed through OpenRouter so it can run on a free model with a paid fallback) does three jobs in one pass — decide if a thread is actually relevant, decide if the poster sounds like they own the thing or is just relaying secondhand, and cluster the recurring points into claims with evidence attached. Everything downstream — the reducer, the support math, the UI — trusts that shaping completely, so getting it right mattered more than almost anything else in the project.

Challenges

Credit discipline was a real design constraint, not an afterthought. SerpApi's free tier is 250 searches a month, and every extra fallback call eats into that. I built a two-stage gather — google_forums first, alone, and only fall back to a second, broader search if the first came back too thin — plus a disk cache for local dev so iterating on the UI costs nothing. Tuning how thin is too thin turned out to be its own ongoing problem: real google_forums responses are noisier than I expected, often dominated by Facebook and LinkedIn results that read as discussion but aren't, which pushes the fallback to fire more often than I originally assumed it would.

Getting the LLM to stop hallucinating specificity was harder than getting it to extract claims at all. Search snippets are cut off mid-sentence — literally ending in "…" — and the model's instinct is to complete the thought plausibly. A plausible completion that the source never actually said is a fabrication wearing a citation. I had to write explicit rules against finishing a snippet's sentence, against reusing a source's distinctive phrasing verbatim, and against letting one source's detail leak into another source's evidence line. None of that is exotic prompt engineering — it's just the unglamorous work of writing rules for every way a language model quietly lies with confidence.

And the last one is the most human: making the board itself feel worth staying on, not just correct. It's easy to build something that's functionally right and viscerally forgettable — an accurate list of claims with checkboxes is not a product anyone wants to spend time in. Getting the support bar to move like something real, making rejection feel like a decision rather than a checkbox, giving the empty state a reason to click something instead of stare at a search box — that's the part with no spec, just taste, and it's the part I kept coming back to rewrite.

Built With

• css3
• model-context-protocol
• node.js
• openrouter
• react
• serpapi
• typescript
• vercel
• vite
• webmcp
• zod