01-site-light
02-site-dark
03-deck-1
04-deck-2
05-deck-3
06-deck-4
07-deck-5
08-deck-6
09-deck-7
10-deck-8

Devpost submission text

Track: Professional Agents · Repo: https://github.com/tusharshah21/FlakeGuard · License: MIT

Form fields

Elevator pitch: Your CI is 85% red but every matrix cell is 94% green. FlakeGuard finds the flakes hiding in that gap, quarantines them, and opens the PR to let them back in when they recover.
Built with: python · amazon-bedrock · claude · strands-agents · sqlite · github-actions · github-api · pygithub · pydantic · pytest · uv · vite · react · typescript · tailwindcss · mermaid · junit-xml
Try it out: the repo, the landing page, and https://github.com/tusharshah21/FlakeGuard/pull/7 (the agent reversing its own quarantine)
Image gallery: local/shots/, 3:2 PNGs of the site and all eight deck slides
Architecture diagram: local/architecture.png

Inspiration

Ask any engineer on a team whose CI has gone unreliable what they do when a build goes red, and the honest answer is: hit re-run. Once a suite fails often enough for reasons nobody believes, people stop reading failures at all, including the real ones. The tests are still running; they have just stopped being evidence.

We wanted to see how bad that gets in a real codebase, so we pointed FlakeGuard at dask/distributed, a large, active, public Python project with a 34-cell test matrix and 89 days of genuinely messy CI history.

The numbers explain why nobody had fixed it. An individual matrix-cell run passes 94.4% of the time. And 162 of 191 scheduled runs (85%) go red. A big matrix is a failure multiplier: each cell flips its own coin, and the whole run fails when any one of them lands wrong. No single cell's dashboard shows a problem worth investigating. Restricted to one cell, 89 days of data surfaces six tests that ever failed. Pooled across all of them, the same window shows nineteen tests failing and recovering over and over.

That is a problem you can only see from above, and nobody has time to look.

What it does

FlakeGuard is an autonomous triage agent for GitHub Actions and pytest. It runs in the background, and it surfaces only when there is a decision to make.

Watches CI. Pulls workflow runs, per-job results and JUnit artifacts into local storage: 704,323 observations across 191 runs and 34 matrix cells in our demonstration window.
Computes the statistics in Python. Per test and per commit: failure probability, a 95% Wilson score interval that stays honest at small sample sizes, how many of the cells a test actually runs in failed, whether failures concentrate on one platform, and whether they begin at a particular commit or a particular date.
Classifies. Flaky, regression, chronic, platform-specific, environment break, or unclear, which is a first-class answer rather than a failure.
Decides in code, not in the prompt. A plain-Python gate holds every verdict against thresholds, sample size, what it already did today, and whether a human has overruled it.
Acts, when the evidence earns it. An issue naming the failing commit and the likely source file; a pull request quarantining a flake so it still runs and reports but cannot fail the suite; a review-queue entry that changes nothing at all when confidence is low.
Reverses itself. A quarantined test that passes consistently for twenty runs gets an automatic pull request removing the quarantine. The agent undoes its own decision without being asked.
Explains. flakeguard explain <test> "why does this only fail on Windows?" hands the same statistics to an agent that sequences its own tools and answers in prose. It is read-only by construction: the action layer is not in its toolset.

The determinism boundary

The model reasons. It never carries a number. Every statistic is computed in Python and pinned into the prompt as ground truth; the model interprets those numbers and never calculates, estimates or rounds its own. We check this mechanically. Every numeric token the model writes must appear verbatim in the evidence it was given, and across 80 classifier outputs, zero numbers were invented.

And the model never decides to write. Classification is a judgement; modifying someone's repository is not. Those are separate functions, and the second one is ordinary Python that a prompt cannot reach.

We did not adopt that split on principle. We measured our way into it. Adding one sentence to one verdict's definition fixed the case it targeted and, at temperature zero, moved an unrelated verdict across the action threshold with self-contradictory reasoning, while a third case started fabricating numbers. We reverted the change and kept the lower score. LLM classification is not locally editable, which is precisely why the thing that decides whether a repository gets touched is not a prompt.

What's real about it

Everything in the repository is reproducible from public data.

A real target: 89 days of dask/distributed CI, 704,323 observations, 191 scheduled runs, 34 matrix cells.
Real artifacts, live: the Issues and Pull Requests tabs of the project's own repository hold what the agent produced. An issue for a Windows-only failure, a quarantine pull request, a review-queue entry it declined to act on, and PR #7, the reversal: one file, one line removed, after the test passed 35 consecutive runs. A human merged the quarantine that preceded it. The agent never merges.
Measured, not asserted: the model is benchmarked against a deterministic threshold classifier on cases built from real data where a single signal misleads. On the straightforward cases both score 4/4. On the hard ones the baseline gets 1 of 4 and the model 2 of 4, a real but modest edge, reported as we found it. One of the two remaining misses may be our own labelling error rather than the model's, and we say so in docs/evaluation.md rather than quietly relabelling the fixture.
Safe by construction: FlakeGuard analyses a repository we do not own and has never written to it. Every pull request targets a dedicated scratch branch; a pull request against main raises before any API call. Writing is opt-in, dry-run is the default, and every run announces its mode on the first line.

The live run

Against this repository's scratch branch, dry_run = false, raw logs in probe-results/live-sweep-*.txt:

step	result
(a) sweep: platform case, chronic flake, ambiguous case	issue #1 (platform_specific @ 0.95); quarantine PR #2 (chronic @ 0.85); review-queue issue #3, one comment (12 runs < `min_runs`, never reached the model). 4 model calls.
(b) the same sweep, same day	all three `none`, "already triaged today". 0 artifacts, 0 model calls. Review queue still one comment.
(c) replay, clock at 2026-08-25	`test_handle_null_partitions_2`, whose real last failure was 2026-08-24, classified chronic @ 0.85 on 156 runs → quarantine PR #4, titled `[REPLAY as of 2026-08-25]`. 2 model calls.
(c) replay, clock at 2026-09-13	35 consecutive clean runs since quarantine ≥ 20 → un-quarantine PR, titled `[REPLAY 2026-08-25 → 2026-09-13]`. The agent reversed itself, on real data, with 0 model calls. First opened as #5 while #4 was unmerged; a human then merged #4 (the agent never merges) and the replay was re-run as #7: one file, one line removed.
(d) a human closes issue #1; sweep with a next-day clock	verdict unchanged, gate says issue, action layer finds the closed issue → `closed_by_human`, nothing recreated, disagreement recorded.

Total for the run: 8 model calls, roughly $0.10. Every PR targets scratch; conftest.py and .flakeguard/quarantine.txt are absent from main and from scratch itself.

Things that behaved differently live than against the in-memory double, all recorded rather than hidden:

Open PRs each carried the quarantine hook (#2 and #4) because each branched off scratch before anything merged. Once a human merged #4 the hook exists exactly once in the merged state, which is what the invariant claims, and the re-run un-quarantine PR #7 is a literal one-line removal. #5 was closed by the operators with a note pointing at #7 and labelled flakeguard-superseded.
closed_by_human blocked the operators' own redo. Closing #5 to replace it looked to the agent like a disagreement, exactly as designed. The escape hatch is a label, flakeguard-superseded, on the closed artifact; without it, a closed FlakeGuard issue or PR is never recreated.
The ledger recorded intent, not outcome, in two places (a blocked issue recorded as issue, a blocked un-quarantine as unquarantine_pr). Both now record what happened.

An operational caveat: issue identity is title-based. Cross-day idempotency ("comment on the existing issue, never open a duplicate") works by exact title match, because a dated title would defeat it. Issue titles therefore carry a date-free [REPLAY] prefix, and editing a FlakeGuard issue's title by hand will cause the next sweep to open a new one. We renamed #1 and #3 live to add the prefix and then verified the matcher resolves them: a next-day sweep commented on renamed #3 rather than opening a second queue, and the closed_by_human check matched renamed #1.

How we built it

Python 3.11, Strands Agents SDK on Amazon Bedrock (Claude Sonnet 4.5), SQLite behind a thin storage interface, PyGithub, Pydantic. Four agents with narrow contracts and structured outputs (classifier, correlation, drafter, and the read-only explorer), plus a statistics module that imports nothing from strands, enforced by a test. The scheduled sweep runs as a GitHub Actions workflow with cached state; a warm run ingests in 19 seconds and costs about six model calls.

The landing page is a separate Vite + React + TypeScript + Tailwind v4 project in site/. Its hero draws one real scheduled run of dask/distributed, all 34 matrix cells with two of them red, from a typed array of the actual run data, so the page cannot drift from the thing it illustrates. The load sequence replays that run: the cells fill in order, the two failures land last, and the word "red" in the headline turns with them.

Challenges

The hardest part was finding out what the data actually was. Our first design assumed GitHub's API exposes test-level results. It does not. The second assumed the target repository retried flaky tests in-process. It does not. What it does do is re-run its entire suite twice daily on an unchanged commit, which turns out to be a far better dataset: many independent samples of identical code, so per-test flakiness becomes a measured probability instead of a heuristic. Four rounds of probing the real repository preceded the first line of ingestion code.

The subtler problem was that our own safety limits did not compose. A cap on how many artifacts a sweep may create sat downstream of the classifier, so an unattended run spent 30 model calls in order to decline 15 actions. The fix was ordering, not a new limit: the cheap check has to run first. Same run, same data, 6 calls instead of 30.

What we learned

LLM classification is not locally editable. We added one sentence to one verdict's definition to fix a case the model was misreading. It fixed that case, 0 out of 10 runs to 10 out of 10. It also moved an unrelated verdict across the action threshold in 4 of 10 runs, with reasoning that contradicted itself, and a third case started writing numbers that appeared nowhere in its evidence: 9 fabrications where there had been none. At temperature 0, same model, same inputs. We reverted the change and kept the worse score.

That is the single most useful thing we learned, and it is why the function deciding whether to touch a repository is ordinary Python rather than a prompt. A classifier you cannot edit locally is not a classifier you can put in charge of writes.

Our own safety limits did not compose. A cap on how many artifacts a sweep may create sat downstream of the classifier, so an unattended run spent 30 model calls in order to decline 15 actions. The fix was ordering, not a new limit: the cheap check has to run first. Same run, same data, 6 calls instead of 30.

Ground truth is harder than the benchmark suggests. One case we scored as a model error may be our labelling error instead. A later reading of the same data supports the model's verdict, not ours. We left the score unrevised and wrote that down, because a fixture relabelled once you know the answer is worth nothing.

Accomplishments we're proud of

A benchmark we did not tune to win. The model beats a deterministic threshold rule 2 to 1 on the hard cases, a real but modest edge, reported exactly as we found it, with both remaining misses explained.
Zero fabricated numbers across 80 classifier outputs, verified mechanically rather than asserted.
An agent that reverses itself. PR #7 removes a quarantine the agent added, after the test passed 35 consecutive runs. One file, one line.
Guardrails that were tested by being violated: a pull request against main raises before any API call, and the safe-by-default write mode exists because a local test once opened a real issue by accident. Both are documented.

What's next

Walking back to the onset commit rather than the latest failing one when correlating a regression. Detecting truncated artifacts. And pointing it at a repository whose maintainers want the pull requests.