TasteTest A/B

An agent that predicts which of two options fits your taste-defined audience. Predictions freeze before the outcome, then get checked against real votes. Powered by Qloo.

Comment

Inspiration

Small businesses and creators make taste decisions on gut feel every week: which special, poster, product name, or event theme will their audience actually pick? Real A/B testing is too slow and too expensive for them.

AI predictions have a second problem: trust. Anyone can claim "our AI knew it" after the outcome is known. TasteTest is built on one rule: a prediction only counts if it was frozen before the outcome existed.

What it does

TasteTest is an agent that turns a gut-feel A/B call into a testable prediction:

  1. Describe your decision in plain language. Type "pho ga vs bun cha for VSA students" and the app splits it into two options and an audience hint, no form-filling.
  2. The agent predicts. Your taste seeds resolve through Qloo's Taste AI into cross-domain affinities. Both options are scored out of 100, and the agent shows its work in six visible steps: resolve seeds, resolve candidates, fetch affinities, score fit, record branches, freeze.
  3. The prediction freezes. Timestamped and SHA-256 hashed. It cannot be edited after the fact.
  4. A live poll collects real votes. Shareable link plus QR code, no login for voters, live progress toward the vote target.
  5. The verdict reports honestly, including misses. Four sections: the verdict, where the branches disagreed, what surprised us, and how the verdict was built. A follow-up chat answers questions grounded only in the frozen verdict data.

Three branches predict side by side: AI-only, Qloo, and the combined Agent. You see exactly what Qloo adds versus a language model guessing alone. A benchmark page keeps three separate tallies: fresh live polls (Track B), a 40-case MovieLens film proxy, and a 40-case Last.fm music proxy. Small polls are labeled directional; they are never presented as statistical proof.

Why Qloo materially improves this: without taste affinities, the agent falls back to text heuristics. We show this openly with a "no data" fallback badge when Qloo has no entity signal (e.g., niche regional chains). Qloo's cross-domain taste graph is the difference between a guess about wording and a prediction about people.

How we built it

  • Frontend: web app hosted on Netlify, design language "Linear calm, Raycast glow," light/dark mode.
  • Qloo Taste AI API called server-side only (key in environment variables, never in client code or the repo): seed resolution via search, affinities via insights.
  • AI-only branch via Replicate, recorded with exact model and version before freezing.
  • Polls and freeze receipts on serverless functions; SHA-256 hash of the frozen prediction shown on every verdict.
  • Two creation modes: Quick test (type → predict, the default) and Record as evidence (the full rigor: track selection, eligibility checks, manual branch recording; the only mode that feeds Track B tallies).

Challenges we ran into

  • Qloo has no data for some entities. Niche chains returned zero affinity for every seed. Instead of silently scoring anyway, we added an explicit fallback badge and an abstain state for ties with no signal.
  • Honesty costs. Reporting "QLOO MISSED" in a verdict feels like anti-marketing, but verifiability is the product. A prediction you can't check is just a claim.
  • The loop needs people. Polls need real voters and hackathon demos don't come with an audience, so we built historical proxies (MovieLens, Last.fm) as a separate, honestly-labeled evidence track.

Accomplishments that we're proud of

  • A complete, working loop (type → predict → freeze → poll → verdict), live with no login wall.

  • Film-proxy evidence: on 11 completed MovieLens cases, Qloo matched the held-out segment outcome 8 times vs 6 for the AI-only branch; in the 4 cases where Qloo and the AI-only branch disagreed, Qloo was right 3 times.

  • First real-audience Track B case closed: TB-201 drew 24 votes, A won 14-10, and both the LLM and Agent branches matched (Qloo abstained). The case is recorded in the closed-case ledger; the accuracy tally counts it once audience membership is verified.

  • An evidence architecture most demos skip: frozen predictions, three separate tallies, misses reported in plain language.

  • The agent's work is visible, not a black box: six prediction steps, per-option score breakdowns, and a verdict that cites its inputs.

What we learned

  • Trust in a prediction product comes from verifiability, not claimed accuracy. Freeze first, then check.
  • Small live polls are directional reality checks; statistical signal needs larger datasets. Labeling them separately keeps the story honest, and honesty turned out to be the differentiator.
  • The music proxy exposed a real limitation: Qloo trailed the LLM baseline 3/6 vs 4/6 on the 6 completed cases, and its entity resolution failed outright on 4 of 10 selected cases (Miley Cyrus, Jennifer Lopez, Glee Cast). We report that rather than hide it.

What's next

  • Running more real-audience Track B cases.
  • A decision log (what did you do with the verdict?) and accumulated taste profiles per audience.

Built With

Share this project:

Updates

Submission history