BehaviorLab

Inspiration

Before a government or NGO rolls out a policy — raising organ donor registration, VAT filing, vaccination, or cutting youth vaping — someone has to answer: which behavioral interventions are most likely to work, how uncertain is that, and how strong is the evidence? Today that answer usually comes from optimism, not evidence. Proposals overstate effects, and nobody plays the skeptic until money is spent. We wanted a tool that researches the evidence, simulates the uncertainty, and then argues back.

What it does

BehaviorLab — Beat the Skeptic turns any policy goal into a short game:

1. Mission — e.g. "53% → 59.6% within 12 months, budget USD 28k". The target is calibrated to what the reviewer-adjusted evidence says the best affordable portfolio can deliver, so it's winnable — but only with a good portfolio.
2. Spend — five evidence-backed interventions, each with a cost. The budget affords two or three, never all five, so you must trade off.
3. Bet — predict how many percentage points you'll really gain. The default is what the proposals claim — accepting it is the optimist's mistake.
4. Reveal — the optimistic result fills the meter… then a red pen strikes each card, the meter shrinks to the reviewer-adjusted result, and 5,000 simulated outcomes drop into a histogram against the target.
5. Score — mission success, calibration (how close your bet was), efficiency (pp per budget unit) and a rank: Optimist, Analyst, Skeptic or Red Pen.

Everything is also exposed to external AI agents through WebMCP tools (generate_scenario, get_scenario, list_interventions, run_simulation, compare_interventions, explain_result).

How we built it

• Evidence discovery — Tavily: searches the web for behavioral-science research and policy experiments relevant to the goal.
• Scenario generation — Nemotron 3 Super (120B-A12B) on Nebius Token Factory: builds a JSON scenario with population, baseline, goal direction and five interventions with low/medium/high effect ranges, assumptions and citations. An intervention is "research-backed" only if it cites a returned source; otherwise it's a hypothesis. Baselines are labelled user / evidence / assumed.
• Probabilistic simulation: a seeded Monte Carlo engine (5,000 draws, triangular distributions) reports median, 5th–95th percentile, worst/best case and probability of improvement, with a 20% diminishing-returns discount for combined nudges.
• Critical review — Nemotron 3 Ultra (550B-A55B) on Nebius Token Factory: a skeptical policy reviewer scoring evidence strength, comparability to the target population and realism, and proposing its own ranges.
• Stack: Next.js 16, React 19, TypeScript, Tailwind CSS 4, WebMCP polyfill, Jest 30. Tavily, Super and Ultra calls are cached server-side (memory + JSON files) so repeat goals cost nothing.

Challenges we ran into

• Preventing false precision: rounding, capping probabilities at "over 99%" and labelling assumed baselines.
• Getting goal direction right (increase vs decrease) so effects move the rate the correct way.
• Making the critique feel like a game moment rather than a wall of text — hence the red-pen reveal.
• Latency of large-model calls, solved with caching and running the review during scenario build.

Accomplishments that we're proud of

• A two-model generator/critic loop where the second Nemotron genuinely disagrees with the first.
• Turning uncertainty and evidence quality into something people feel, not just read.
• An app that humans and AI agents can both drive via WebMCP.

What we learned

Optimistic effect sizes are the default failure mode of policy design — and of LLMs. Pairing a fast generator with a larger skeptical reviewer, grounded in retrieved evidence, produces far more honest plans.

What's next for BehaviorLab

• Localised evidence and costs (e.g. Nepal in NPR, already supported) for more countries
• Population segments and interaction effects between interventions
• Shareable mission reports for policy teams

Built With

• ai
• jest
• monte-carlo
• nebius
• next.js
• node.js
• nvidia-nemotron
• react
• tailwindcss
• tavily
• typescript
• webmcp