Quribo

Validate ideas, messages, and brands before building. Quribo turns a brief into a reviewable plan, runs synthetic stakeholder interviews, and reports what people value, object to, or find confusing.

Comment

Inspiration

Before building something, how do you know whether anyone would care? Real user research is slow, expensive, and often arrives after a team has already committed to a direction. The question was whether a system of AI agents — configured as different personas and stakeholders — could surface the kind of reactions, objections, and motivations a concept test is supposed to explore, fast enough to inform an early decision rather than confirm a late one.

Two things shaped the design heavily:

  1. Recent research (Maier et al., 2025) shows that directly asking LLM personas for Likert ratings produces unrealistic distributions, while eliciting free text and mapping it to scores via semantic similarity preserves both qualitative and survey-style signal. This pushed the design to separate participant elicitation from later analysis — participants talk naturally, a separate pass codes the evidence.
  2. The conviction that synthetic responses are not evidence about real populations. If the product was going to be honest, every finding had to trace back to a transcript quote, and every report had to carry its method limits.

What it does

Quribo turns an open, messy brief — typed or dictated — into a reviewable validation plan, runs synthetic stakeholder interviews, and produces an evidence-linked report.

  1. Brief → the user explains a product, message, or brand idea in natural language (voice supported).
  2. Understanding review → the system extracts a validation map: what should be contrasted, and against whom.
  3. Variations → targets, persona profiles, stakeholder roles, and situation contexts — each item marked as user-provided, inferred, or model-suggested.
  4. Combinations → a reviewable combination strategy instead of a blind cartesian product.
  5. Approve → nothing simulates until the user signs off.
  6. Simulate → a bounded batch of coverage-driven synthetic interviews, each with a hidden conversation director.
  7. Analyze & synthesize → a separate pass extracts semantic observations tied to exact transcript quotes, then synthesizes report-shaped findings with confidence and evidence counts.

Every finding links back to an observation, which links back to a message, which links back to a quote.

How we built it

The stack is a single TypeScript codebase deployed on Vercel, with four layers:

  • Domain schemas shared across the whole system — they define the contract between frontend, API, and storage, so every layer speaks the same language.
  • Planning engine — takes a raw brief and runs extraction, critique, and reconciliation against Google Gemini to produce a structured validation design. A compiler then turns the approved design into executable scenarios with agent specs.
  • API + persistence — an Express server running as a Vercel Serverless Function, backed by Amazon DynamoDB for all persistence. The simulation runners (batch execution, semantic analysis, synthesis) live here.
  • React 19 client — a product wizard for end users and an admin back-office for inspecting every artifact in a run, with a design-token system, dark mode, and full E2E test coverage.

The evidence chain is enforced in code, not just in documentation: every report finding must cite a semantic observation, which must cite a transcript quote. The synthesis step rejects any finding that references a non-existent observation, and a deterministic fallback keeps the chain inspectable even when a model call fails. Method limits are always injected — the synthesis cannot persist without them.

Why DynamoDB? The data is nested JSON documents accessed by ID plus a time-ordered list of artifacts per run — exactly DynamoDB's native access pattern. A single-table design with one GSI handles all queries. Filesystem persistence was never an option on serverless, and DynamoDB's on-demand scaling means there's no database to manage.

Challenges we ran into

  • Personas kept blurring into stakeholder roles. Generated designs produced personas that read like soft role descriptions and roles that duplicated the same actor. The fix was prompt engineering plus an explicit invariant — same person can hold many roles; same role can be held by many personas — rather than a schema change.
  • The first concept became the product architecture. A single reference case (LocalPass) was silently defining the whole system. The course correction was to make it a generic validation engine with LocalPass as just one example brief.
  • Measurement validity. The hardest intellectual problem: when is a synthetic response a useful proxy and when is it a model artifact? The decision was to never present derived scores as observed human behavior, and to keep elicitation and analysis as separate stages.
  • Migrating persistence to serverless under a deadline. The planning engine originally wrote debug snapshots to disk — none of that survives on Vercel serverless. The engine was refactored to a pure function and all persistence moved behind a store interface, so API routes didn't change.
  • Keeping the codebase maintainable. The main app component had grown past 2,000 lines with duplicated logic between the product and debug interfaces. It was split into focused screens, reusable primitives, and shared helpers before adding features on top.

Accomplishments that we're proud of

  • The evidence chain works end-to-end. A reviewer can open any report finding, follow it to a semantic observation, and land on the exact transcript quote that supports it. That traceability is enforced by the system, not bolted on after the fact.
  • Honesty is built into the system, not a disclaimer. The synthesis cannot persist without method limits; the pipeline rejects findings that cite no real observation; copy across both languages avoids words like "validated," "representative," or "users prove."
  • A reviewable contract before any simulation runs. The validation-planning layer — understanding, variations, combinations, approval — exists as a first-class, editable artifact. Every item carries provenance: user-provided, inferred, model-suggested, or model-critiqued.
  • A one-click pipeline that degrades gracefully. Compile → batch → analyze → synthesize runs from a single endpoint, and a deterministic fallback keeps the chain inspectable even when the synthesis model call fails.
  • Dark mode and a full design-token system from day one — zero hardcoded colors outside the token file, which forced the design system to be complete before feature UI piled on top.
  • The back-office replaced a frozen debug workspace with a proper inspector/operator for the whole run lifecycle, from brief to synthesis.

What we learned

  • The most valuable part isn't the simulation — it's the validation-planning layer that decides what to test and against whom before any conversation runs. If that contract is weak, the simulation is theater.
  • Traceability is a feature, not a polish step. Forcing every finding to cite a quote changed how the synthesis had to behave, and made the report dramatically more trustworthy.
  • Honesty about limits is a differentiator. The demo never claims market validation, representativeness, or real population evidence — and that constraint made the report more useful, not less.
  • Synthetic panels are exploratory, not confirmatory. They're good at surfacing hypotheses, objections, and language worth testing with real people; they're bad at producing numbers that should drive a decision on their own.

What's next for Quribo

  • More validation templates — message/campaign testing, brand perception, adoption barriers, and an interactive evidence report — currently exist as stubs; only product concept testing is complete.
  • Interactive evidence workspace: conversational Q&A over the bounded study record, where every answer cites supporting observations and the assistant admits when the study can't answer a question.
  • Multi-agent group discussions for research questions involving influence, negotiation, diffusion, or conflict — currently every interview is independent.
  • Provider and prompt-variant matrix so the same scenario can be contrasted across models and prompt framings, instead of the current Google-only, sequential execution.
  • Branching conversation trees with cached resumption, rather than linear interviews.
  • Validation work comparing synthetic results against real panels, to learn where the method is a useful proxy and where it is a model artifact.
  • Production hardening — background execution, durable queues, and proper access control. The hackathon slice runs synchronously and behind a shared password.
  • A complementary monitoring subscription that reuses a completed study as a contextual baseline for interpreting external signals over time.

Built With

Share this project:

Updates