ExperimentationAI

Inspiration

Product and growth teams thrive on rapid iteration, but the reality of A/B testing is anything but fast. We saw a major bottleneck in the manual, time-consuming process of monitoring live experiments — the daily logins, the tedious data interpretation, and the slow feedback loops. We were inspired to solve this post-launch challenge and build a tool that would let teams move at the speed of their ideas.

What it does

ExperimentationAI is an intelligent agent that automates the entire post-launch phase of A/B testing. It connects to your existing A/B testing platform and data lake to monitor and analyze results in real-time, delivering value through:

Automated Slack Updates: Proactive alerts that flag significant results and declare winners, eliminating the need for manual oversight.
An Executive Dashboard: A real-time, at-a-glance view of all ongoing experiments with summarized results and data visualizations.
Data-Driven Recommendations: Actionable insights that go beyond simple data presentation to suggest the next best steps.

Ultimately, the vision is for the agent to close the loop entirely — automatically rolling out winning variants and even suggesting the next logical experiment to run.

How we built it

We chose a modern, modular stack to bring this to life. Agent orchestration is powered by LangGraph, allowing us to model the monitoring workflow as a stateful graph. The core reasoning and analysis is driven by Google Gemini. We built integrations for GrowthBook, AWS Athena, and SQLite to reflect real-world data environments, and used Datadog for production-grade observability.

Challenges we ran into

Our biggest challenge was scoping. The potential for an experimentation automation tool is enormous, and we had to be disciplined in focusing on the post-launch phase where we felt the most acute pain. This also required us to generate a realistic mock dataset from scratch — a necessary step to validate our agent's logic before working with live data.

Accomplishments that we're proud of

Experimentation is the primary mechanism by which consumer technology businesses improve themselves — and we've built a way for experimentation to improve itself. Beyond the technical achievement, we solved a problem every member of our team faces daily. It is not a solution looking for a problem; it is a solution built by the people who have the problem.

What we learned

Our cross-functional team of a PM, a growth strategist, an engineer, and a business stakeholder came together at this hackathon, and for some, it was their first time using AI coding tools. We learned that a diverse team deeply connected to the problem can build truly impactful solutions — and that modern AI tools are making it possible for everyone to be a builder.

What's next for ExperimentationAI

Feed the engine: Expand the dataset to improve the agent's analysis and confidence.
Close the loop automatically: Build the functionality for the agent to not just declare a winner, but to roll it out to 100% of users without manual intervention.
Recommend what's next: Evolve the agent from a monitoring tool into a true continuous growth engine that actively proposes the next experiment to run.