Claim-Verified Analytics Memos

Inspiration

Analysis memos are often written by hand or by AI, and the numbers in them can be wrong, badly rounded, or made up. Teams then make decisions on those numbers. We wanted a memo where every figure can be traced back to its calculation.

What it does

Claim-Verified Analytics Memos turns a two-group CSV, such as an A/B test or a pricing experiment, into a short analysis memo and checks every number in it.

• Drop in a CSV or load a sample. The app picks the outcome column and the two groups.
• It computes the group averages, the difference, a 95% confidence interval, and whether the result is significant.
• It writes a memo from those figures.
• A verifier matches every number in the memo back to its calculation and flags any figure it can't trace.
• Hover a number in the memo to see its source. Copy or download the memo with its verification results.
• A "simulate an edited number" toggle shows the verifier catching a changed figure.

How we built it

The statistics engine is plain JavaScript and uses Welch's two-sample t-test. The confidence interval uses a t critical value for small samples and a normal value for larger ones. The memo is generated from a template, and the verifier checks each number in the text against the computed claims. The interface is one HTML page with CSS and vanilla JavaScript, with no framework or build step. It is hosted on Freebuff Sites as a static deployment.

The product itself uses no AI model. The statistics, memo text, and verification are all deterministic code. The project was built with the Buffy coding agent.

Challenges we ran into

Keeping the verifier honest was the hardest part. A naive check that searches for numbers as substrings would pass "40.0" when "40.00" was expected, so the verifier matches whole number tokens. A unit test covers that near-miss case.

Choosing the confidence-interval method for small groups also mattered. A normal approximation gives a narrower interval than a t-distribution, so we use the t-distribution.

Accomplishments we're proud of

• The statistics are reproducible. The sample results match a separate Python calculation.
• The verifier catches both an edited number and a near-miss number, and both cases have tests.
• All 13 unit tests pass, and the live site runs the full workflow with no console errors.

What we learned

Separating the calculation from the writing makes results easier to trust. A verifier is only useful if it fails on realistic mistakes, not just obvious ones.

What's next

• Support for more than two groups and for confounders.
• An optional AI writing step, where the verifier still checks every figure in its output.
• Uploading real datasets and publishing an example for each analysis type.

Built With

• css
• freebuff
• html
• javascript
• sites
• welch's