TrustCheck

Does this money claim hold up? Paste a post, transcript or link and see the evidence, not the hype.

Comment

Inspiration

Social feeds are full of money claims: "This coin 10x'd in a month," "guaranteed 5% a week," "the Fed just cut rates." Scam emails look more like real banks every year. Most people can't check these claims, and when they ask an AI chatbot they get a confident paragraph with no sources that can be just as wrong as the original post.

For the FNB Challenge we wanted to build something that helps people before they send money and right after something goes wrong. It shows its evidence and never tells anyone what to do with their money.

What it does

Core: Claim check. Paste a social post, video transcript, article link or news headline. TrustCheck:

  • Splits it into claims and tags each one as a fact, a numeric claim, a forecast or an opinion.
  • Does the math itself. "5% every week" becomes the yearly return it implies (over 1,000%), so you can see how extreme a claim really is.
  • Finds evidence and ranks every source. Tier 1 is primary sources (SEC, Federal Reserve, BLS, IRS, exchanges, company investor-relations sites), Tier 2 is established news (Reuters, AP, Bloomberg, WSJ…), Tier 3 is everything else.
  • Pulls live data when it helps, marked with a "Live data" badge: crypto prices (CoinGecko) and official economic data from FRED work out of the box; stock quotes (Finnhub) and SEC filings switch on with a free API key and an SEC contact string.
  • Gives each claim a verdict: Supported, Mixed, Contradicted or Can't confirm.
  • Never calls a prediction true or false. Instead it checks whether any official source or trusted outlet makes the same prediction ("Others predict this" or "No reliable backing"). An extraordinary prediction nobody reliable makes, like "+100% tomorrow", or one pushed with "buy now" pressure, comes back "Doesn't hold up". Opinions are marked "Can't be verified".
  • Answers "Should I buy X?" with the evidence instead of advice: it checks any prediction in the message, adds today's market picture for that asset, and shows a bottom line built by code (the claim's verdict, the current price, and which way sourced factors lean). It never says buy or sell.
  • Flags pressure language like "guaranteed," "can't lose," "get in now."
  • Gives an overall label (Holds up / Needs checking / Doesn't hold up), an evidence gauge showing how decisive and which direction, plain-language reasons, and a full evidence trail with verbatim quotes and links.

Beyond single claims:

  • Email check: upload a suspicious .eml file. TrustCheck flags link text that doesn't match the real link, look-alike domains, failed SPF/DMARC checks, disguised attachments like invoice.pdf.exe, and urgency wording. Then it checks the email's claims. Attachments and links are never opened, and personal info is redacted.
  • ScamSOS: "I think I was scammed." Five quick questions produce a calm plan for now, today and this week, plus a case summary you can paste to your bank. Every step links to official FTC or FBI IC3 guidance. Nothing is stored.
  • "What this implies" calculators: for savings, fee, interest and loan claims, shows what the numbers really add up to, using only the numbers the post states.
  • News alerts: a watchlist of tickers or topics that surfaces headlines from reliable outlets only (official SEC filings can be added with an SEC contact string), with browser notifications. One click checks any headline.
  • Market briefing: sourced context on a stock or topic, with a code-computed balance of the drivers for and against a prediction. It shows direction only, never price targets or buy/sell calls.

TrustCheck judges how well a claim is supported. It never gives financial advice.

How we built it

Our core rule is AI reads and explains; deterministic code decides.

input → fetch URL → extract claims (Gemini) → math check (code) → find evidence (Gemini + Google Search grounding, live data tools) → tier sources (code lookup) → classify stance (Gemini) → decide verdict (code) → hype flags (code) → aggregate (code) → write explanations (Gemini + advice lint) → result

  • Gemini on Google Cloud Vertex AI does the language work: pulling out claims, judging whether a source supports or contradicts a claim, choosing data tools, and writing the explanations. Evidence comes from Gemini's Google Search grounding.
  • Plain TypeScript decides everything that matters. Source tiers come from an editable JSON lookup table, not the model. Verdicts come from tier-weighted evidence scores (Tier 1 = 3, Tier 2 = 2, Tier 3 = 0.5, with Tier 3 capped so blogs can never settle a claim on their own). Each domain counts once.
  • Live data tools use Gemini function calling, but the model only requests tools. Code checks each request against an allow-list, validates the arguments, builds the URLs and makes the call itself. The model never sees an API key. Results go through the same stance and verdict rules as any other evidence. Sources are CoinGecko, Finnhub, SEC EDGAR and FRED.
  • Guardrails on the model:
    • Quotes have to appear word for word in the source.
    • Cited evidence IDs have to exist.
    • Every number in a market briefing has to appear in the retrieved text.
    • Explanations are checked for advice wording.
    • If the model's output fails these checks, we fall back to fixed template text.
  • ScamSOS is rules-first. A playbook file decides the steps, using sources we read and checked ourselves. The model may only reword them, and a validator rejects any rewording that adds links, numbers, or words like "refund," "guarantee" or "recover."
  • Backend: Node.js, Express, zod, Helmet, rate limiting, SSRF-safe URL fetching, mailparser for email files, optional Postgres for check history.
  • Frontend: React, Vite and Tailwind CSS, with live progress streamed over Server-Sent Events, dark mode, and Auth0 login for saved history.
  • Testing: 224 unit tests with Vitest, covering tiering, math, verdicts, aggregation, validators, SSRF, email signals, ScamSOS, calculators and more. There's also an eval harness for real public claims.

Challenges we ran into

  • Keeping the LLM honest. Models invent quotes, mix up which source said what, and make up numbers. We built verbatim-quote checks, source-matching checks (a source's text has to share real words with the claim) and number checks. Our first live market run caught the model citing the wrong source for a fact, which is why the source-matching check exists.
  • Dates. Without being told today's date, the model treated recent real events as "forecasts." Passing the date fixed it.
  • Noisy news feeds. Our first news run returned a Nike story for NVDA and pages of options quotes. We added topic aliases, whole-word matching and filters for quote-page junk.
  • Defining "holds up." Turning evidence into a label meant making explicit calls: what counts as a central claim, how mixed evidence is weighed, and how much hype should downgrade a result. Our first gauge even showed "Insufficient" for claims that strong sources clearly refuted, so we redesigned it to show how decisive the evidence is and which way it points.
  • Staying out of advice territory, especially for the market briefing. We built sourced context and a direction-only balance, and deliberately left out buy/sell/hold calls and price targets.
  • Respecting data sources. Our SEC tool only runs when it identifies itself as the SEC's fair-access policy asks; we show only headlines and links from news feeds, and we dropped data sources that sit behind bot checks.
  • Moving to Vertex AI mid-build. We switched from an API key to Google Cloud credentials so every Gemini call runs through one shared client, and learned to handle expired logins and timeouts on slower calls.

Accomplishments that we're proud of

  • An AI fact-checker where the AI can't decide the verdict. Every label traces back to specific sources and a rule in a config file.
  • A full journey from before (claim check, email check, news alerts) to after (ScamSOS), all built on the same evidence-first approach.
  • An email checker that catches a fake-bank phishing email, with a look-alike link, failed authentication and an invoice.pdf.exe attachment, without ever opening a link or attachment.
  • A ScamSOS plan that's calm, specific and fully sourced, and never promises refunds.
  • 224 passing tests on the rules that matter.

What we learned

LLMs are great at reading and terrible at being judges. Splitting those jobs made TrustCheck more accurate and easier to explain. We also learned:

  • how to use Vertex AI search grounding and function calling safely
  • how to stream progress over SSE
  • how much careful wording it takes for "not financial advice" to be true of the product itself, not just its footer
  • that every "first live run" taught us a new validation rule

What's next for TrustCheck

  • A browser extension that checks posts right on X, Reddit and TikTok
  • Video and audio input with automatic transcription
  • Published accuracy numbers from a larger set of real public claims
  • Email and SMS alerts for the watchlist
  • A partner version for banks: flag risky claims or payees before a customer sends money.

Built With

Share this project:

Updates

Submission history