A/B Testing Readout Agent

Inspiration

I'm a Data Scientist actively interviewing for senior roles, and the pitch I keep making in interviews is that experimentation rigor is the actual job, not dashboards, not p-values pulled out of context, but knowing when a result is trustworthy enough to act on. Meanwhile I kept seeing the same failure mode on real teams: someone opens an A/B test dashboard, sees a green metric, and ships, without checking whether the sample size supported the claim, whether the groups were even split the way they were supposed to be, or whether the test had enough power to detect anything real in the first place. Most teams don't have a dedicated statistician catching this before a decision gets made. I wanted to build the thing that stands in for that person, not an LLM guessing at significance, but a tool that actually runs the correct test, checks the assumptions a statistician would check, and only recommends shipping when the result earns it.

What it does

You point it at a CSV with a variant column and an outcome column (binary conversions or a continuous metric), and it:

• Auto-detects whether the outcome is binary or continuous and runs the appropriate test, a two-proportion z-test for conversions, or a Welch's/pooled t-test (chosen via Levene's test) for continuous metrics.
• Reports the effect size, a two-sided confidence interval, and an achieved-power estimate against an 80% planning target.
• Runs pitfall checks a human statistician would run before trusting the number: minimum sample size, group-size imbalance, extreme conversion rates, and a sample-ratio-mismatch (SRM) chi-square test, SRM is one of the most common silent causes of a biased result, and it's rarely checked by hand.
• Produces a conservative ship / do-not-ship / inconclusive recommendation, where "inconclusive" wins over a significant p-value if the test was underpowered or the sample ratio looks broken. A significant result on a shaky foundation is not treated as a green light.
• Hands the structured result to GPT-5.6 for a two-step narration: one call drafts a plain-English readout for a non-technical stakeholder, and a second, independent call reviews that draft against the raw statistics and prior experiment history, correcting it if the prose contradicts a warning or states a number that isn't actually in the data.
• Logs every analysis to a persistent local history, so you can see how today's result compares to prior tests.
• Ships with a small Streamlit front end so none of this requires touching a terminal.

The confidence interval for the binary case, for reference, is the standard Wald interval on the difference in proportions:

$$ (\hat{p}T - \hat{p}_C) \pm z{1-\alpha/2}\sqrt{\frac{\hat{p}_T(1-\hat{p}_T)}{n_T} + \frac{\hat{p}_C(1-\hat{p}_C)}{n_C}} $$

How we built it

Solo, across a handful of evenings around a full-time job, using Codex end-to-end. Codex wrote the initial scaffolding, the CLI entrypoint, the synthetic-data generator for testing, and the first pass at the statistical core. From there, every extension was a scoped Codex prompt that I reviewed before accepting: the sample-ratio-mismatch check and its unit test, the persistent JSONL experiment log, the two-step GPT-5.6 draft-and-review narration loop, and the Streamlit UI wrapping it all. Codex also helped fix the rougher edges near the end, an interrupted git init that left a stale lock file, and a .gitignore that needed to actually exclude private project notes and __pycache__ before the first commit.

The statistical logic itself was deliberately kept free of any API calls, so it could be unit-tested and hand-verified in isolation from the LLM layer, that separation mattered more here than in a typical hackathon project, because the entire value proposition rests on the numbers being right, not just the prose around them.

Challenges we ran into

The biggest one was trusting but verifying Codex's statistics before building anything on top of them. I ran the analysis engine against known synthetic scenarios, a clear binary effect, a null effect, deliberately underpowered small samples, a continuous outcome with a known true mean difference, and cross-checked the p-values and confidence intervals by hand against manual scipy/statsmodels calls before trusting the code further. That caught nothing wrong, which was reassuring, but also confirmed it needed doing: the one thing this project can't afford to get quietly wrong is the math.

Second was making the "ship" recommendation actually conservative instead of naive. Early on, a significant p-value on a tiny, badly-underpowered sample would have still read as "ship", I had to explicitly force the recommendation logic to treat underpowering and sample-ratio mismatch as overriding a significant result, not just as a footnote warning next to it. I verified this by deliberately engineering a small-sample run that hit p < 0.05 by chance and confirming the tool still called it inconclusive.

Third, more mundane but time-consuming: an OpenAI API key that returned a 401 "insufficient permissions" error because it had been generated as a restricted key without the model.request scope, and a git repository that silently never finished initializing, both eating time I didn't have much of against the deadline.

Accomplishments that we're proud of

Getting the pitfall-detection logic to actually change the recommendation, not just print a warning next to a decision nobody reads carefully, is the part I'm most proud of, it's the difference between a tool that looks rigorous and one that actually stops you from shipping on noise. Close behind that is the self-review loop on the GPT-5.6 side: it's a genuine second, independent pass that checks the first model's own explanation against the source data, not a single prompt dressed up to look agentic.

What we learned

That "observed power", power calculated from your own sample's effect size after the fact, is a legitimate but genuinely debated diagnostic, useful as an underpowering signal but not the same thing as a pre-registered power analysis, and worth being precise about which one you're presenting. And, less statistically and more practically, that verifying a Codex-generated statistics module against synthetic ground truth before building narration on top of it is not optional busywork, it's the actual engineering work on a project like this.

What's next for A/B Testing Readout Agent

Sequential-testing and peeking detection is the natural next pitfall to add, this version checks a single completed dataset, but a lot of real-world bias comes from checking results early and repeatedly rather than from a single bad snapshot. I'd also like to support multiple simultaneous metrics with multiple-comparisons correction, since real experiments rarely have just one outcome. And I want to close the loop on the original motivation for building this: pick a concrete before/after metric, accuracy at flagging known pitfalls against a labeled set of synthetic experiments, or time saved versus a manual readout, so the project can honestly back up a resume bullet, not just a demo.

Built With

• agentic
• codex
• experimentation
• statistics