Inspiration
A small consumer brand rarely has a research budget. When a founder must pick one of three product concepts, the default "research" is asking a chatbot — which answers confidently and invents the rest. We wanted to know how much of that confidence can be replaced with real cultural-affinity evidence and deterministic code, and how honest a machine-assisted decision can be made to look on one screen.
What it does
Describe your market and audience, enter exactly three competing concepts, and set how much each criterion matters to you. The app then:
- Retrieves cultural-affinity evidence from the Qloo API
- Scores the concepts with transparent deterministic code
- Has an LLM explain the result — and a second pass critique that explanation within strict limits
- Re-runs the ranking across up to 1,000 sampled weightings, reported as "Top-ranked in X% of tested preference scenarios."
Every evidence record carries its provenance; every limitation of a run is shown in the report. It is a concept-screening aid: cultural affinity is not a guarantee of purchase intent, and the app never claims to forecast sales.
How Consentiens Culturae Is Different
A normal search engine helps users find information. An LLM-powered search tool helps them interpret it. Consentiens Culturae aims to help small consumer brands compare product concepts and justify a decision.
The Difference
| Approach | Purpose | Output |
|---|---|---|
| Web search | Find information | Links, articles, trends |
| LLM search | Summarize and interpret evidence | Answers and recommendations |
| Consentiens Culturae | Evaluate competing concepts systematically | Evidence-linked rankings, constraints, and sensitivity analysis |
How It Works
The system combines Qloo cultural-affinity signals with validated web evidence. Deterministic logic applies weighted criteria and hard constraints, while AI assists with evidence retrieval, explanation, and critique without controlling scores or rankings.
Sensitivity analysis tests whether the leading concept changes when user priorities shift. Reports preserve evidence provenance and acknowledge uncertainty, including when the available evidence is insufficient.
Why It Matters
Rather than simply asking which concept seems best, users can examine why it ranks first, what evidence supports it, and how stable the ranking is when priorities change.
Consentiens Culturae does not predict sales, and cultural affinity does not prove purchase intent. Its value lies in combining an auditable decision process with evidence users can inspect.
Similar features could be built into an LLM search product. The differentiation is the integrated decision workflow, whose value must be demonstrated through comparisons against ordinary search and LLM research using the same briefs.
How we built it
Every run follows the same four stages: evidence retrieval, deterministic scoring, bounded critique, and sensitivity analysis. We chose this structure so that each result can be traced back to its inputs and scoring logic.
Architecture
The backend uses Python 3.11, FastAPI, Pydantic v2, and SQLAlchemy. PostgreSQL handles production data, while SQLite is used during local development. A typed state machine, split between states.py and transitions.py, works with a deterministic coordinator to manage the workflow.
The frontend uses Next.js 16 and runs on Vercel. The backend is deployed on Render.
The application is a single, modular system. We haven't introduced microservices, message queues, or a vector database because the current workflow doesn't require them. This keeps the codebase and deployment easier to manage.
Evidence retrieval
The application uses three Qloo endpoints: /search, /tags, and /insights. Each run has a limit of 10 API calls, a 10-second timeout, and one retry with backoff for HTTP 429 responses.
An LLM proposes a retrieval plan based on the user's brief. Before any request is sent, the application checks each proposed call against an endpoint allowlist and the remaining API budget. If the plan fails validation, the system uses a deterministic fallback.
Search results also need to match the intended category. Live searches are restricted to urn:entity:place, preventing unrelated books and media from entering the evidence set. Category terms are resolved to valid place tags through a controlled lookup process. The application rejects identifiers that haven't been validated, even if the model proposes them.
For market-specific results, the shared market call uses Qloo's signal.location parameter. This tailors affinity results to the user's market without requiring another API call.
Scoring
Once retrieval is complete, deterministic code calculates the rankings using four weighted criteria: cultural affinity, differentiation, market context, and evidence coverage.
Hard constraints are checked separately from the weighted score. A concept that violates a known constraint is marked INFEASIBLE. If the information needed to check a constraint is missing, the system leaves it unresolved and displays a notice. The concept can still be ranked, but the report makes the missing information visible.
Each run records a scoring_version, identifying the version of the scoring logic used. We tested reproducibility by running the same brief 10 times and confirming that it produced the same result.
Bounded critique
The language model helps explain the rankings and identify problems in the analysis. Its output passes through code-based validation before it reaches the report.
The workflow uses three roles:
- Analyst: Explains the ranking using structured results and actual evidence IDs. Its response must conform to a defined schema.
- Critic: Checks claims against the retrieved evidence, using a separate evidence map for each concept. It reports issues through fixed codes and permits at most one revision.
- Suggester: Identifies potentially relevant concept attributes. An attribute is labelled stated only when its wording appears verbatim in the original concept description. Paraphrased attributes are automatically classified as inferred.
Each role has a deterministic fallback for model failures or invalid output. The final report also identifies which sections were written by a model and which were generated from templates.
Sensitivity analysis
A concept's position in the ranking depends partly on how the founder/user weights the evaluation criteria. Sensitivity analysis shows how the ranking changes when those preferences change.
The system generates up to 1,000 seeded weight vectors and applies the same scoring logic to the frozen evidence. It records how often each concept ranks first, the range of positions it occupies, and which criteria influence changes in the winning concept.
The report uses the phrase "Top-ranked in X% of tested preference scenarios." This describes the ranking across the scenarios tested. It isn't a probability of success and makes no claim about future sales or product-market fit.
Persistence and operational safeguards
Five database tables record the runs and their results:
decision_runsstores run status and configuration.evidence_recordsstores retrieved evidence and its provenance.workflow_eventsrecords state transitions and workflow activity.score_snapshotsstores calculated scores and rankings.critic_findingsrecords issues found during critique.
Incomplete runs can be resumed or deleted. A per-engine result cache and client-side single-flight handling reduce duplicate work when requests or polling overlap. A run limiter also helps control model usage against the available quota.
Testing and verification
The project currently has 265 offline tests covering scoring, constraints, missing data, reproducibility, adapter behaviour using fixtures, state transitions, critic schemas, and adversarial cases.
The test suite runs without network access or API keys. It also refuses to load a local .env file, so offline tests cannot silently rely on locally configured credentials.
Live Qloo behaviour is verified separately, with the observed API details recorded in an internal reference. This separates tests that use fixtures from checks performed against the live service.
Fixture data is always labelled, and each report identifies the source of the evidence it used. If a run uses demo fixtures, the report says so rather than presenting the results as live Qloo data.
The system keeps evidence retrieval, scoring, and explanation under separate controls. Qloo supplies the cultural-affinity evidence, code determines the rankings, and the language model explains and critiques the results within defined limits. Founders can then review the evidence, see how their preferences affect the outcome, and identify what still needs to be checked before testing a concept with real customers.
Problems We Found and Fixed
1. Qloo searches returned the wrong entity types.
Our first live searches returned a mix of books, media titles, and other entities. Using those results would have introduced irrelevant evidence into the analysis. We restricted live searches to types=urn:entity:place and verified the behavior against the live API. Fixture mode was left unchanged.
2. Some category names had no matching Qloo tags.
During a live run, the category computing devices did not match any place tag in Qloo. The resolver now checks category:place first, then genre:place. If neither produces a match, the report records the limitation and the system skips the market call rather than retrieving unrelated evidence. The planner cannot invent entity IDs, so it cannot fill the gap with a fabricated ID.
3. The critic lacked concept-specific citation context.
The critic checked whether analyst claims referenced valid evidence IDs, but it received only a flat list of IDs. As a result, it flagged some valid citations because it could not determine which concept the evidence belonged to. We changed the input to include a per-concept evidence map, allowing the critic to check each claim against the evidence available for that concept. This reduced false flags without changing the limit of one revision.
4. Paraphrases were incorrectly labelled as stated attributes.
The attribute suggester sometimes marked paraphrased information as something a concept had explicitly stated. A stated label now requires an exact, case-insensitive match against the concept's own name, description, or attributes. If the text does not match, the basis is changed to inferred, the unsupported quote is removed, and the report records the change in a note.
5. Concurrent polling exhausted the database connection pool.
During a live run, the frontend aborted after 15 seconds while multiple poll requests continued to arrive. The 15-slot SQLAlchemy connection pool was exhausted, and the results page returned a 503 error. We added a per-engine result cache so repeated polls can use one inexpensive query. The client now allows only one poll at a time, and when the database is busy, the application asks the client to retry rather than returning a generic error. The response makes clear that the run's results have not been lost.
6. Automatic .env loading affected offline tests.
When we added automatic .env loading for local development, live settings—including real API keys—were also loaded by the offline test suite. Two tests then failed without an obvious explanation. The loader now skips .env when pytest is running. The application continues to load the file during normal local development, while the test suite runs without depending on those live settings.
What we learned
Several of our problems came from shortcuts that seemed reasonable during implementation. The attribute suggester turned paraphrases into things a concept had supposedly “stated.” The constraint gate treated missing information as a violation, making entire concepts INFEASIBLE for reasons that could not be verified. We changed the gate so a known violation makes a concept infeasible, while a missing attribute produces a visible notice. That distinction took more work to implement, but it gave us a result we could actually justify.
We also found that asking a model to check its own work more carefully was not enough. The critic was flagging valid citations because it could not see which evidence belonged to which concept. Giving it a per-concept evidence map fixed the missing context. Similarly, requiring an exact match against a concept's own text stopped the suggester from presenting paraphrases as direct statements. In both cases, the improvement came from changing the information and rules around the model, not simply asking it to behave better.
Qloo's 10 call limit also influenced the architecture in useful ways. We found that signal.location could adjust affinity for the target market without an additional call. We also built a resolver ladder to handle category names that did not map cleanly to Qloo tags, rather than repeatedly guessing. The same constraint shaped the critique process: one critic pass and, at most, one revision. With free-tier limits and runs that take five to fifteen minutes, extra model calls need a clear purpose.
Testing the live API changed our assumptions more than we expected. Place searches returned books and media until we restricted the entity type. The category computing devices matched no suitable place tag, which led us to add explicit handling for unresolved categories. We recorded these findings manually in an append-only notes file after live probes. That record matters because the API's actual behavior did not always match our expectations, and assumptions that seemed reliable during planning could become outdated quickly.
We learned that error messages are part of the product's trustworthiness, too. When concurrent polling exhausted the database connection pool, users needed a clear explanation of what had happened and what to do next. A retryable database-busy response is more useful than a generic server error, provided the system can support the claim that the run's results remain intact. We applied the same principle to model failures: each LLM role has a deterministic fallback, so the report can still be produced when a model is unavailable. It also identifies which sections were generated from templates instead.
Our tests are what allow us to make these claims with confidence. We check determinism by running the same brief ten times and comparing the results. Tests also enforce the boundary that the LLM cannot write scores. The offline suite covers scoring, constraints, missing data, state transitions, and other critical behavior without requiring network access or API keys. When automatic .env loading caused two tests to fail, it exposed a problem in our own development setup. Skipping .env during pytest restored the separation between local configuration and offline verification.
The broader lesson was practical: reliable behavior came from explicit rules, relevant evidence, and tests—not from assuming a model would make the right judgment. The system is more defensible when it can show what it knows, identify what is missing, and explain where its output came from.
What's next for Consetiens-Culturae
The roadmap is a direction, not a fixed schedule. Progress will depend on evaluation results, technical readiness, and funding for larger infrastructure changes. For the hackathon, the system will remain a single deployable service. The priority is to make it reliable before making it bigger.
1. Fix the Immediate Bottlenecks
LLM-heavy runs currently take five to fifteen minutes. Because they execute within the request cycle, a single run can occupy a worker, delay concurrent users, and fail before completion. The first step is to move execution into a durable background workflow, with persisted state, idempotent operations, and reliable recovery—without changing the scoring logic or splitting the application into services.
Accounts and saved briefs come next, building on the existing PostgreSQL records for runs, evidence, scores, workflow events, and critic findings. This makes previous decisions accessible and prepares each run to become an independent unit of work.
2. Prove Quality Before Adding Complexity
The current offline suite has 265 tests covering scoring, constraints, missing data, reproducibility, state transitions, and adversarial cases. The next step is to expand the versioned evaluation benchmark with held-out briefs and human-reviewed judgments.
Evaluation will cover ranking agreement, evidence quality and traceability, constraint handling, sensitivity analysis, critic accuracy, reproducibility, latency, API cost, and report usefulness. Correctness, provenance, and constraint handling are mandatory release gates; recommendation quality and operational reliability follow.
Latency must also be measured by pipeline stage before performance targets are set. Optimizations should reduce cost and delay without weakening evidence quality or deterministic guarantees.
3. Improve Evidence Gathering and Critique
Qloo cultural-affinity data will remain a core input, supplemented by validated web evidence. Cultural affinity is not treated as purchase intent, and the system will not turn cultural signals into unsupported sales forecasts.
Models may propose retrieval plans, but deterministic checks will enforce query budgets, valid entities, and retrieval constraints. Some API capacity will be reserved for filling evidence gaps. The critic will separately assess citation integrity and challenge the substance of a concept, with any revisions bounded and validated.
A lightweight open-weight model may eventually run in shadow mode and be compared with the current critic and human judgments. Until that evaluation supports a change, it will not replace the existing approach. When evidence is insufficient or materially conflicting, the system should qualify its conclusions or abstain rather than guess.
4. Establish Safer Releases
Changes currently deploy directly to Render and Vercel. A staging environment with automated build, test, and delivery pipelines will help catch regressions before they reach the demo or pilot. Deterministic fixtures will support repeatable tests, while integration checks will cover live dependencies where practical.
Automated gates will be followed by human release approval, informed by test results, benchmark comparisons, and operational measurements.
5. Split Services Only When Justified
The longer-term architecture may separate the pipeline into retrieval, deterministic scoring, and model-assisted critique, coordinated by a deterministic workflow engine. Retrieval would own Qloo and web calls, rate limits, and evidence validation; scoring would enforce calculations and hard constraints; critique would handle claim checking and bounded revisions.
If funding and measured demand justify the move, AWS Lambda or an equivalent managed platform could host suitable workloads, with durable queues between stages. Until then, a modular monolith with clear interfaces and data ownership is the better fit. Microservices will be introduced only when their reliability or scaling benefits outweigh their cost and operational complexity.
6. Preserve Reproducibility
Scoring versions already change when behavior changes: the label moved from v0.1 to v0.2 after the hard-constraint gate changed, with tests checking where the version is recorded. Future changes will follow the same rule.
The approach will extend to retrieval policies, model configurations, prompts, benchmarks, and report schemas. Published decisions should retain immutable snapshots and version manifests. Where exact replay is no longer possible, the system must label a reconstruction clearly. Material changes should trigger targeted replay and comparison, with human approval before revised results are published.
7. Validate With a Controlled Pilot
Once the mandatory gates pass, a small pilot with explicitly provisioned users will test whether the reports genuinely help people compare concepts. Evaluation will cover task completion, recommendation quality, trust in evidence, sensitivity-analysis usefulness, latency, cost, and user feedback.
Tenant isolation, access revocation, retention, and verified deletion will be part of the pilot's operational requirements. Approved feedback can inform the versioned benchmark. A formal review of results, costs, technical debt, and user feedback will determine what comes next.
The guiding principle is simple: improve reliability and evidence quality first, validate the product with users next, and expand the architecture only when results and funding justify it. The goal is a decision engine whose recommendations are traceable, rankings reproducible, and uncertainty explicit—not a larger system for its own sake.
Built With
- fastapi
- nextjs
- postgresql
- python
- qloo
- render
- vercel