TasteCase

Find out whose taste your AI restaurant assistant forgot, and turn the cases you care about into tests you replay before every release.

Comment

Find out whose taste your AI restaurant assistant forgot, and turn the cases you care about into tests you replay before every release.

Inspiration

Picture a hotel concierge who sends an indie rock fan and a pop fan to different places. Then the concierge is replaced. The new one sounds just as friendly, but every guest gets the same three steakhouses. Nobody complains, so nobody notices.

AI assistants that recommend restaurants have the same problem. Teams change prompts and models all the time, and a new version can quietly stop paying attention to who's asking while every answer still reads well. Format checks and LLM graders tell you an answer looks right. They don't tell you which guests' recommendations moved. We wanted a QA tool that looks for those cases on purpose and keeps a person in charge of what counts as a regression.

What it does

TasteCase is for the engineering and QA team behind a restaurant assistant. The team brings its own restaurant catalog, the task its assistant answers and the practical constraints, such as city, party size and dietary needs. Then TasteCase:

  1. Creates two fictional guests with the same constraints and different public interests, and asks Qloo how each guest would rank the team's own restaurants.
  2. Uses code to find restaurant pairs whose order flips between the two guests. A free model can only choose from those code-checked flips, so it can't invent taste facts.
  3. Sends both guests' requests to the team's assistant through a local runner, so the assistant's API keys stay on the team's machine. Each request repeats, which separates real differences from noise.
  4. Shows a review card with Qloo's rankings beside the assistant's answers. A person accepts, edits, rejects or defers it and writes down why.
  5. Publishes accepted cases into a suite that can't change, compares two named assistant versions against it, freezes the report and exports a test pack that replays locally without calling Qloo again.

TasteCase never grades taste automatically. A Qloo ranking difference stays advisory until a person decides otherwise, and missing Qloo coverage counts as unavailable evidence, never as a dislike.

How we used Qloo

Qloo decides which cases are worth a person's time. The backend runs Qloo's official MCP harness, @qloo/qloo-harness, against the hackathon API. It maps each restaurant and guest interest to a Qloo entity with qloo_describe, and the owner confirms each match. Then qloo_rank ranks the team's eligible restaurants through /v2/insights for each guest. The order flips between those rankings are the only contrasts the model may choose from, and a smaller follow-up ranking checks the chosen pair. Without Qloo, a model would have to make up a cultural contrast. With it, every case starts from observed ranking evidence on the team's real catalog.

Each review card also shows why Qloo ranked things that way. Rankings request feature.explainability, and TasteCase keeps the place tags Qloo already returns from /entities. A reviewer sees each restaurant's affinity score and the gap between them, how much each guest interest contributed, and the cuisine, ambience and style tags only one side has. None of this adds a request. A small gap is shown as a weak contrast and never turned into a verdict.

We wrote up what we learned building on the harness and API, with concrete suggestions, in FEEDBACK.md: https://github.com/CryptoZephyr/tastecase/blob/main/FEEDBACK.md

How we built it

  • A Next.js and React Aria Components web app on Vercel, with all API access behind one typed, same-origin bridge.
  • A Fastify and Node API and discovery worker on Render, with a Postgres-backed queue, leases and checkpoints, so a restart reuses saved evidence.
  • Supabase for sign-in, Postgres with row-level security and private storage.
  • A local runner CLI that runs the team's assistant and uploads checksummed results.
  • Free models through OpenRouter, with zero price caps and no paid fallback. Dots 3 Note Preview picks the contrast on the backend, and NVIDIA Nemotron 3.5 Super VL is the runner's default assistant.

Challenges we ran into

  • Qloo rate limits. Our first pilots hit HTTP 429. The harness re-resolved every entity ID at once and retried internally, so one ranking could turn into nine requests. Qloo support confirmed a limit of 5 requests per second. We now batch lookups, cap each workflow at two HTTP requests, turn off hidden retries and space requests 250 ms apart. The next pilot made six requests, and all returned 200.
  • Free model churn. Our first free model was delisted mid-hackathon and another was rate limited upstream. One model spent about 760 of 771 output tokens on hidden reasoning and returned an empty answer until we turned reasoning off. We tested three free alternatives with the worker's real request settings and moved the selector to Dots 3 Note Preview, which returned valid picks 5 of 5. Every model we tried picked the first listed contrast, so the selector adds little judgment so far.
  • An intermittent hosted failure. Two production runs stopped on a Qloo ranking step with PROVIDER_INVALID_DATA, and we couldn't reproduce it locally. The worker now logs a safe description of the failure, with schema paths and no Qloo values. It hasn't come back in later runs, and the cause is still unknown.
  • A bias we caught in our own wording. The follow-up question listed the restaurant pair in one guest's Qloo order for both guests, and the assistant copied that order. It now lists the pair in catalog order and says the order isn't a preference. The public sample's follow-up answers came from the old wording, and the docs say so.
  • Keeping evidence honest. Partial coverage, unknown outcomes and provider failures stay visible instead of being turned into passes or failures.

Accomplishments that we're proud of

  • A full hosted journey: discovery with live Qloo rankings, human review, suite publishing, an 8 of 8 version comparison, a frozen report and a test pack download.
  • A public sample from a real hosted run on an 8-restaurant catalog, reviewed and accepted by the owner. Qloo ranks Dirt Candy and Momofuku Noodle Bar in opposite orders for a Radiohead fan and a Taylor Swift fan, with small affinity gaps of 0.004 and 0.019. All eight Qloo requests returned 200 and all six assistant answers came back. The assistant gave the Radiohead fan the same list in the same order twice, with Momofuku ahead of Dirt Candy, and gave the Taylor Swift fan only Carbone, Le Bernardin and Peter Luger, twice. Both answers said nothing in the catalog matched the guest's interest. That's the kind of personalization gap TasteCase is built to surface.
  • Hosted resilience checks on production. After a Render cold start, the API answered its first health check in about 42 seconds and later requests in under a second. A queued comparison kept all eight attempts through a service restart. Cancelling a comparison or a discovery job stopped new work, with no further Qloo or model calls. A downloaded test pack passed its checksum check and replayed locally with zero Qloo calls. With bounded retries in the runner, a replay got 5 of 8 answers from the free model, and TasteCase reported the other 3 as unavailable rather than passing or failing them.
  • A first human review. The owner rated all seven blinded pilot cards useful, from both the Qloo-guided method and a plain LLM, so the review didn't separate the two. The reasons pointed at the assistant's uneven personalization.
  • More than 700 automated tests, including real Postgres isolation and concurrency checks.

What we learned

Cultural ranking evidence works best as a way to choose what to test, not as an answer key. Treating gaps as unknown, keeping the model on a short leash and leaving the final call to a person made the product much easier to trust.

What's next

  • Run a study with more than one reviewer and pair-specific assistant answers to measure whether Qloo-guided discovery finds more useful cases than a plain LLM. We haven't shown that yet, and we don't claim it.
  • Retry assistant answers that time out during discovery.
  • Verify objective-rule failures in the hosted browser journey.
  • Find the cause of the intermittent PROVIDER_INVALID_DATA.
  • Support catalogs larger than the current ten-option ranking limit.

Built With

Share this project:

Updates

Submission history