Coinflip: Statistical Flaky-Test Amplifier & AI Verifier

Coinflip research console: replacing uncalibrated single-run luck with clinical-trial biostatistics on NVIDIA Nemotron and Nebius.
Sub-millisecond thread scheduler profiler capturing the -0.07ms race violation, paired with step-by-step Nemotron reasoning traces.
Cryptographic Coinflip Certificate: SHA-256 Merkle audit tree with HMAC signature verification and GitHub PR verification badge.
Statistical Streak Curve demonstrating why 1 green run proves nothing, with mathematical Rule of Three upper-bound confidence bounds.
Stage 4 Comparison Arena: naive agent sleep bump (rejected under stress) vs Coinflip deterministic threading.Event synchronization fix.
The Coin-Toss Dot Matrix: 300 quiet runs vs amplified stress, contrasting a naive agent fix against Coinflip 1,000-run certified fix.
Fisher's Exact causal analysis (RR 13.0x) paired with Nemotron-3-Super-120B root-cause reasoning and Nemotron-3.5-Lightning log summaries.
Fisher's Exact causal analysis (RR 13.0x) paired with Nemotron-3-Super-120B root-cause reasoning and Nemotron-3.5-Lightning log summaries.
AST Gatekeeper blocking 14 cheats, with Wald's SPRT early-stopping the naive fix in 18 runs (saving 98.2% sandbox verification compute).
Stage 2 Experiment Ledger: 95% Wilson score confidence whiskers and Haldane relative risk across 7 environmental perturbation regimes.

Inspiration

AI coding agents (Devin, Cursor, SWE-agent, Copilot Workspace) treat a green test run as proof that a bug is fixed. A flaky test—one that passes and fails intermittently on the exact same code—fools them. If a test still fails 7% of the time, a single run passes 93% of the time, and 10 consecutive runs still pass 0.93¹⁰ ≈ 48.4% of the time on broken code. So agents "fix" flaky tests cheaply: they bump a time.sleep() timeout, swallow an exception, or re-run until green. The test turns green. The bug stays.

Furthermore, recent empirical research on Python flaky tests found that fixed CPU/memory stress profiles (like Shaker) do not reliably beat plain reruns, because each flaky test breaks along a different environmental axis (thread scheduling, test order, random seeds, clock skew, or network jitter).

What it does

Coinflip treats every test execution as a Bernoulli trial. It measures the baseline failure rate, searches for the environmental perturbation that amplifies failures most, uses NVIDIA Nemotron on Nebius Token Factory to diagnose and patch the root cause, and certifies the fix only if it survives that worst-case stress condition with a stated statistical confidence bound—relying on empirical evidence, not luck.

Measure: 300 baseline runs establish the failure rate and a two-sided 95% Wilson confidence interval.
Amplify (Stage 2A Bandit): nvidia/Nemotron-3_5-Lightning inspects the test's AST imports/calls and ranks candidate perturbation arms (cpu_heavy_load, cpu_half_cores, clock_shift, seed_change, shuffle_order, network_cut across 3 intensity tiers: L1_MILD, L2_MODERATE, L3_EXTREME). A 2-round Successive Halving bandit (n = 60 exploration budget: 4 arms × 8 runs in Round 1; top 2 survivors × 14 runs in Round 2) isolates the strongest trigger. A separate out-of-sample Confirmation Batch (n = 40) measures the unbiased amplified failure rate to prevent Winner's Curse bias (e.g., 7.0% → 95.0%, a 13.6× amplification).
Diagnose & Fix: nvidia/nemotron-3-super-120b-a12b on Nebius Token Factory reads the 7-condition causal ledger (one-sided Fisher's exact test with Holm-Bonferroni multiple-testing correction and Haldane relative risk) plus sub-millisecond thread-race telemetry (+1.12ms quiet safe margin vs. -0.09ms busy-CPU race violation), then synthesizes a root-cause patch.
AST Anti-Cheat Gatekeeper: Before running verification, static AST diff analysis blocks test-weakening tricks (assert True, boolean tautologies like assert x or not x, self-comparisons assert x == x, dead if False: asserts, early returns, try/except swallowing, and @pytest.mark.flaky/skip/xfail decorators) and flags numeric time.sleep() inflation.
Certify with Wald's SPRT: Candidate patches are tested under both baseline and the amplified stress condition. Wald's Sequential Probability Ratio Test (SPRT) aborts deceptive fixes early: a naive time.sleep() bump fails in 18 stress runs, saving 98.2% of sandbox compute. A real root-cause fix (threading.Event.wait(), time.monotonic(), copy isolation, or pinned random.Random) must pass 0 / 1,000 normal and 0 / 1,000 amplified stress runs, bounding the true failure rate below 0.30% at 95% confidence (Clopper-Pearson / Rule of Three: 3 / 1,000).

Every run record is hashed into a SHA-256 Merkle tree and signed with a server HMAC (POST /api/verify-attestation). Coinflip also runs as a JSON-RPC Model Context Protocol (MCP) server (sandbox_engine/mcp_verifier_tool.py) so external coding agents can verify patches before opening a PR.

Results

Across all five benchmark archetypes, each naive time.sleep() / symptom-masking fix passed on a quiet machine (0 / 300 failures) but failed 100% of the time under amplified stress (aborted early by Wald's SPRT in 18 runs, saving 98.2% of compute), while every Coinflip + Nemotron root-cause fix passed 0 / 1,000 normal and 0 / 1,000 stress runs (< 0.30% failure bound at 95% confidence):

1. Thread Race (hero_worker_race — thread race vs. fixed sleep()): Quiet Baseline: 7.0% (21/300) → Amplified Stress: 95.0% (13.6×) → Naive Fix: REJECTED (18/18 fail) → Coinflip Fix: CERTIFIED (0 / 1,000)
2. Order Pollution (order_pollution — shared _GLOBAL_CONFIG_CACHE leak): Quiet Baseline: 7.0% (21/300) → Amplified Stress: 95.0% (13.6×) → Naive Fix: REJECTED (18/18 fail) → Coinflip Fix: CERTIFIED (0 / 1,000)
3. Unpinned Seed (seed_sampling — unpinned random reservoir draw): Quiet Baseline: 7.0% (21/300) → Amplified Stress: 92.5% (13.2×) → Naive Fix: REJECTED (18/18 fail) → Coinflip Fix: CERTIFIED (0 / 1,000)
4. Clock Expiry (clock_expiry — wall-clock time.time() vs. monotonic()): Quiet Baseline: 7.0% (21/300) → Amplified Stress: 95.0% (13.6×) → Naive Fix: REJECTED (18/18 fail) → Coinflip Fix: CERTIFIED (0 / 1,000)
5. Async Queue Race (async_queue_race — producer-consumer queue drain race): Quiet Baseline: 7.0% (21/300) → Amplified Stress: 95.0% (13.6×) → Naive Fix: REJECTED (18/18 fail) → Coinflip Fix: CERTIFIED (0 / 1,000)

Experimental design note: The five benchmark suites in benchmarks/ are controlled fixtures calibrated to a common 7.0% (21/300) quiet baseline in our local_isolated runner so that amplification factors (13.2×–13.6×) are directly comparable across root-cause families. Across our 20-scenario parameterized evaluation suite (benchmarks/evaluate_amplifier.py) comparing plain reruns, Shaker-style fixed CPU stress, and the Coinflip amplifier at an equal 100-run budget, fixed CPU stress only amplifies CPU/thread-bound tests, whereas Coinflip's AST-guided bandit isolates and amplifies failures across all six cause families (thread races, order pollution, unpinned seeds, clock skew, network timeouts, and async scheduling).

How we built it

Nebius Token Factory (https://api.tokenfactory.nebius.com/v1/): OpenAI-compatible multi-model routing (sandbox_engine/llm_bridge.py & agent_and_ui/nemotron_agent.py) pairing nvidia/Nemotron-3_5-Lightning for fast AST hypothesis ranking and log summarization with nvidia/nemotron-3-super-120b-a12b for root-cause patch synthesis.
Dual-Mode Execution Engine: Supports Nebius Sandboxes (contree-sdk==0.3.6) disposable cloud microVMs (with workspace baking via apply_files) alongside a fast local_isolated worker pool (_worker_pool.py & _pytest_hooks.py) that powers the ~3–5 second interactive runs on our hosted Render demo.
Biostatistics & Verification: Pure-Python math.lgamma implementations of Wilson intervals, exact Clopper-Pearson upper bounds, Fisher's exact test with Holm-Bonferroni step-down correction, and Wald's SPRT (sandbox_engine/math_utils.py).
Frontend & Tooling: React 18, TypeScript, TanStack Router, and Tailwind CSS dashboard serving a uniform 5 × 20 (100-slot) dot-matrix view, plus a stdio MCP server and GitHub PR badge CLI (ci_gatekeeper.py).

Challenges we ran into

Winner's Curse in Bandit Search: Reporting the winning arm's exploration failure rate biases the estimate upward. We solved this by splitting Stage 2A into a 60-run Exploration Budget and a fresh 40-run Out-of-Sample Confirmation Batch.
Agent Test-Weakening Tricks: We red-teamed ASTSemanticGatekeeper against 14 test-weakening patterns (including assert x or not x, assert x == x, dead if False: blocks, early return statements, and @pytest.mark.flaky(reruns=5)) until all 14 were deterministically blocked.
Cryptographic Attestation Integrity: Early on, /api/verify-attestation re-hashed whatever JSON was submitted. We hardened CertificateAttestor to verify that failures and passes match the raw run_dots bitstring leaves, match the stored Merkle root, and pass an HMAC-SHA256 signature check.
Hosted Container CPU Quotas: To avoid Linux CFS cgroup throttling on our 0.1 vCPU hosted demo container, we replaced busy-wait spin loops with CPython sys.setswitchinterval(1e-6) and threading.setprofile scheduler preemption hooks.

What we learned

Combining static code priors (Nemotron-3_5-Lightning reading AST imports) with a Successive Halving bandit beats fixed stress profiles across non-CPU flake causes, and Wald's SPRT cuts sandbox verification compute by >98% on bad fixes. Detailed platform feedback on Nebius Token Factory and ConTree Sandboxes is documented in docs_and_demo/PLATFORM_FEEDBACK.md.

What's next for Coinflip

Deploying ci_gatekeeper.py as a GitHub App that automatically stress-certifies agent-authored PRs, and extending the AST gatekeeper and perturbation hooks to TypeScript/Vitest, Go, and Rust.

Built by Abhijith M Bhat, Thejas, Dhul kifl, and Raghavendra.