Flaky

• 

  The dashboard you walk into once you enter Flaky

• 

  Flaky diagnosing an error case

• 

Inspiration

Flaky tests waste engineering time because most CI tools only report that a test failed—they do not explain why it behaves nondeterministically. We built Flaky to turn inconsistent test results into actionable root-cause investigations.

What it does

Flaky analyzes CI run history to detect tests that produce different outcomes on the same commit. It combines test source code, error output, timing, execution order, and parallel-run metadata to identify likely causes such as race conditions, shared state, external services, and resource contention.

It then uses Gemini to explain the evidence, recommend a concrete code fix, and—when necessary—generate a ready-to-paste quarantine PR description with measurable exit criteria. The dashboard also tracks each test’s flakiness score and trend across commits.

How we built it

We built Flaky as a single Next.js App Router application using TypeScript and Tailwind CSS. It runs locally with seeded, realistic CI data and requires no external database setup.

Gemini 3.5 Flash powers root-cause classification, fix generation, and quarantine reasoning. Model calls are isolated in lib/ai.ts, use structured JSON responses, and fail over to Gemini 3.1 Flash-Lite during temporary overloads. A deterministic demo mode keeps the project presentable without an API key.

Challenges we ran into

The hardest challenge was making AI output reliable enough for an engineering workflow. Diagnoses needed to be specific, evidence-based, and machine-readable rather than generic suggestions. We added structured response schemas, careful prompts, bounded requests, validation, and visible model attribution.

We also needed realistic flakiness scoring: an ordinary failure is not necessarily flaky. Flaky specifically looks for different outcomes from repeated runs of the same commit.

Accomplishments that we're proud of

We delivered a complete end-to-end investigation flow: detection, evidence review, AI diagnosis, suggested code changes, trend analysis, quarantine documentation, CI-log importing, and live simulated runs.

Every AI verdict exposes its reasoning and supporting signals. The app is responsive, locally installable, and seeded with multiple realistic failure patterns alongside stable tests.

What we learned

We learned that trustworthy developer AI needs more than a confident answer. It needs traceable evidence, constrained output, honest confidence levels, and a useful next action. We also learned that execution metadata—especially commit identity, test order, duration, and parallelism—can be as important as the error message itself.

What's next for Flaky

Next, we would integrate directly with GitHub Actions, ingest JUnit reports automatically, create suggested-fix pull requests, and track whether proposed fixes actually reduce flakiness over time. We would also add team ownership, notifications, and support for additional CI providers.

Built With

• gemini3.5
• local-json-storage
• next.js
• tailwind-css
• typescript