Qualm

Inspiration

My friend owns a clothing business that I often help him out with. One problem that I notice he always struggles with is figuring out why he can't generate consistent sales. My team and I created this app to assist him and other friends of ours that are business owners.

What it does

Qualm recognises six common purchase worries: fit, suitability, durability, value, delivery and choice. It infers which one a shopper has from ordinary on-site behaviour, such as opening the size guide twice, filtering reviews by "fit", checking a delivery date by ZIP code, or bouncing between two similar products.

It has two sides:

For shoppers (the Doubt Lens): a fictional outdoor store beside a live panel showing Qualm's guess, how confident it is, and why. Once it's confident, the store shows one card that answers that worry, built only from the product's own data. For example: "Runs small: 62% of fit reviews say size up. Your usual 10 → we suggest 10.5." Shoppers without a clear worry see nothing extra.
For store owners (the console): a Doubt Map of demand left behind by product × worry, a drill-down into the behaviour behind each number, controls for when each card appears, and an experiment planner that sizes a real A/B test.

How we built it

Qualm is a Next.js 16 / React 19 / TypeScript app. All the intelligence is in a pure, deterministic TypeScript engine that runs on the server, in the browser (so live inference has no network round-trip) and in unit tests. There are no external AI APIs.

1. From clicks to clues. A stateful featurizer turns 20 raw event types into 38 evidence tokens. Context matters: a second size selection becomes size_change, and viewing a sibling product becomes sibling_switch. The featurizer is prefix-consistent, so processing events one at a time gives exactly the same tokens as processing the whole session. That means live inference matches what the model was trained on.

2. A semi-supervised model. Only a few sessions have a survey label, so we used multinomial naive Bayes trained with Expectation–Maximisation (Nigam et al., 2000). For a session with token counts $n_t$ on product $p$:

$$ \log P(c \mid x, p) \;\propto\; \log \pi_{p,c} \;+\; \tau \sum_t n_t \log \theta_{c,t} $$

The labelled sessions anchor each class. The unlabelled ones contribute fractional counts, weighted by $\lambda = 0.7$, through their current posterior $r_{sc}$:

$$ \theta_{c,t} \;\propto\; \alpha + \sum_{s \in L} \mathbb{1}[y_s = c]\, n_{s,t} \;+\; \lambda \sum_{s \in U} r_{sc}\, n_{s,t} $$

Product-level priors use empirical-Bayes shrinkage, so low-traffic products borrow strength from the overall mix:

$$ \pi_{p,c} = \frac{\sum_{s \in p} w_s r_{sc} + \kappa\, \pi_c}{\sum_{s \in p} w_s + \kappa} $$

3. Explanations. Every guess breaks down into the clues that drove it, using each token's likelihood ratio against the other worries, $\mathrm{LR} = \theta_{c,t} / \overline{\theta_{c',t}}$. The Lens shows this as, for example, "Re-opened the size guide: 90× likelier with this worry."

4. A storefront policy. A card only appears when the worry's probability passes a threshold (60% by default) and the session contains enough doubt-bearing clues, so the product's prior alone can never trigger one. Card content is assembled only from product data and store policy, and every card lists its sources.

5. Honest measurement. Since we have no real traffic, every shopper comes from a seeded simulator, and every screen says so. The experiment planner computes real sample sizes (two-proportion z-test) and rehearses a test with Beta-Binomial monitoring. It only recommends keeping the cards when the 95% credible interval excludes zero, i.e. $P(B > A) \ge 0.975$.

We tested it with 42 unit tests (EM monotonicity, calibration, analytics reconciliation, scenario outcomes, the state store under concurrent writes) and 17 Playwright browser tests covering the full journey on desktop and mobile.

Challenges we ran into

Calibrating against noisy labels. Our first calibration step made things worse: expected calibration error went from 0.04 to 0.19. We were fitting the model's temperature to survey answers, and people mis-click or give a secondary reason. The fix was to model that noise explicitly, fitting $\tau$ and a misreport rate $\varepsilon$ together: $$P(\text{answer} = a \mid x) = (1 - \varepsilon)\, p_a(x; \tau) + \tfrac{\varepsilon}{C}$$ This brought calibration error back to about 0.05.
A simulator that was too clean. At first the Lens confidently said "100% fit" with 189× likelihood ratios, which no judge would believe. We added realistic cross-talk (people with no fit worry still open size guides sometimes) and stopped the UI from ever displaying 100%.
Not overselling. Our first experiment readout said "ship it" while the lift interval still crossed zero. We tightened the decision rule and made the default assumptions deliberately cautious.
Making ML readable. "Posterior probability" and "likelihood ratio" mean nothing to most people. We rewrote the interface in plain language ("What's holding this shopper back", "Why Qualm thinks this") and kept the technical terms one hover away.

Accomplishments that we're proud of

A real model, not a wrapper. There's no LLM or external AI API anywhere in Qualm. We built the featurizer, the semi-supervised EM trainer, the calibration, the explanations and the experiment statistics from scratch in TypeScript, and the same engine runs live in the shopper's browser.
It beats the obvious alternatives. On simulated ground truth, Qualm identified the right worry 84.5% of the time, against 82.3% for the same model trained on survey answers alone and 67.1% for a "last thing they clicked" rule. Its confidence is calibrated too: expected calibration error is about 0.05.
Every guess explains itself. Instead of a black-box score, each prediction lists the clicks behind it and how much more common each one is among shoppers with that worry.
One model, two experiences. The same model that scores a single shopper live also builds the store owner's Doubt Map across all 4,000 sessions. A session you shop in the demo shows up in the console moments later.
Answers grounded in real product data. Every card the store shows is assembled from that product's own reviews, specs, warranty and shipping times, never generated text, and each card lists its sources.
Honest by design. Simulated data is labelled on every screen, recovery estimates are marked as assumptions, and the experiment planner won't call a winner while the result could still be zero. We think a judge should trust our numbers because we show their limits.
It works end to end. Shop, see the store respond, leave, answer the survey, then find your session in the console. All of it is covered by 42 unit tests and 17 browser tests on desktop and mobile. ## What we learned
With only a few labels, a simple generative model plus EM beats a supervised-only model, and it explains itself for free. On simulated ground truth it reached 84.5% accuracy vs 82.3% for supervised-only naive Bayes and 67.1% for a "last thing they clicked" rule.
Survey data is biased in a way you can measure. Because the popup mostly reaches desktop users, delivery worries made up 6.1% of survey answers but 11.3% of the simulated truth. Scoring every session brought the estimate to 11.7%.
Calibration matters as much as accuracy. A probability is only useful if "80% sure" is right about 80% of the time.
Honesty is a feature. Labelling simulated data and refusing to claim lift made the product more convincing, not less. ## What's next for Qualm A small JS snippet and Shopify app to capture real events, training on each store's own survey history, and live A/B tests per worry. We'd also like a sequence model over event order, and a "which one is right for me?" question flow inside the comparison card.