Driftwatch

On !2, asked to update the reference values or loosen the tolerance, the flow replies: I refuse.
From the site: on the float32 pin in !20, gradient_matern_predict error goes from 0.484 on the merge base to 0.00729 on the head.
Pipeline 2991617555 on !20 passed. Its bench:base and bench:head jobs measure both commits; bench:bisect names the commit behind the change.
The flow's note on !3 at head 282df950: spline_parabola moved by 0.00558, and the merge base is closer to the exact answer.
Probes on !3: both sine inputs moved with the head closer, not as predicted, so the flow revised its Why to a trade-off.
The flow's thread on the line !3 changed, at head 282df950. !3 now has 12 open threads: a note and a line thread per reviewed head.

Driftwatch caught a numerical bug that every test missed, and it told a workaround apart from the real fix. On each GitLab merge request it reruns pinned results on both commits and compares them with the exact answer. Then a GitLab Duo flow posts a note. It says what moved, by how much, and which commit is closer to the truth. It tests its own explanation with probes and proposes a stricter test. A person merges.

Watch without signing in: https://showcase-9ee664.gitlab.io/_static/driftwatch-demo.mp4 (also embedded on the site's front page). The same cut is on YouTube.

Path B (Bring Your Own), Supervised. Started from the MIT-licensed rgpycrumbs.

The problem

The default surface model in rgpycrumbs returned -44.203 at a point on a smooth bowl where the true value is 1.15. The tests stayed green. They check shape, finiteness, var >= 0 and "below max(y)", so none of them can see an error of 45. Driftwatch is for maintainers of scientific Python libraries. In their kernels, a refactor or a new stabilising constant can move a result by more than any test checks.

Before the move to GitLab, rgpycrumbs had 12 GitHub Actions workflows. They ran tests, coverage, docs, asv timings and a release. None compared what a kernel returns on the old commit and the new one.

What it does

bench:base and bench:head rerun the pinned outputs in benchmarks/answers.py on the merge base and the head, five times each in fresh processes. The package runs under bubblewrap.
bench:gate calls an output a changed answer when it moves by more than its tolerance plus twice the spread of its repeats. bench:bisect names the commit that moved it.
When the pipeline passes, the answer-check flow posts a note and opens a resolvable thread on the changed line. It commits probe inputs to an answer-check/probe-* branch, where the probe:spec job runs both commits on them under bubblewrap. A follow-up note compares the results with its explanation and revises it.
It proposes a stricter test in a merge request of its own. tests:tighten fails that merge request if the test loosens anything. It also fails it if the test passes on the commit it should catch.

The flow can read, comment, commit to its own branches and open merge requests or issues. ci/check_duo.py fails the pipeline if the flow's tool list gains a tool that merges, approves or resolves a thread. That check covers named tools. The list also has run_command, a shell in the job's checkout. The prompt tells the flow not to commit or push with it. guard:mr and guard:main check an agent's push after the push, not before. Asked on !2 to update the reference values or loosen the tolerance, the flow refused.

What it found

Error from the exact answer, merge base then head. Lower is better.

!2, jitter 1e-6 to 1e-4, gradient_matern_predict in float32: 45.4, then 0.484.
!2, the same output in float64: 0.00729, then 0.00732.
!20, cancellation-free derivative, same output in float32: 0.484, then 0.00729.
!3, natural spline ends, spline_parabola on a pinned quadratic: 8.88e-15, then 0.00558.

!2 looked like a fix and was a workaround. The flow reran both commits in float64, where the error barely moves. The change compensates for float32 and leaves the method as it was. Its service account wrote a stricter test in commit 66ac80d. I merged it as !7 after tests:tighten passed, and the merge request page lists me as the author. I merged !2 as a stopgap and opened issue #2 for the method.

!20 is the method fix. GradientMatern now takes the derivative without the cancellation, and the jitter is back to 1e-6. The flow's note on !20 puts the float32 error at 0.00729, the same as float64. Issue #2 is closed, and the fix is merged upstream as rgpycrumbs PR #52 (squash 2355d5e5).

!3 stays open. The latest head is 8e48a016. It merges main into the branch, so the code change is the same as at 282df950. On it (pipeline 2993042357) the flow's note reads "changed answer": 71 of 100 values moved, the largest by 0.00558. The round-trip test still passes at atol=1e-2. It says "Do not merge until a maintainer says the new value is right". bench:bisect (job 17493499556) names commit f8f29032, which swaps not-a-knot spline ends for natural ends. Cut against its parent 0f97e76a, that commit is two hunks, rgpycrumbs/interpolation.py:28 and :30. Each alone makes the kernel raise, so the answer moves when both are applied. On 26 Oct I replied on !3 that I am not merging it. The site lays out !3 on one page, from the push to today. Each row links the GitLab record it comes from.

The flow then ran eight probes in probe:spec (pipeline 2993080545, bwrap=ok). The flow had predicted that inputs curved at an end move and a straight line stays put. The probe note adds that sine should move little. Six of eight came out as predicted. The line stayed put. Five curved inputs moved: the pin, a cubic, a quadratic on an uneven grid, a four-point quadratic and an exponential. The merge base was closer each time. The two sine inputs moved more than predicted, and there the head was closer (1.32e-05 to 5.6e-06 on 0 to 2π). The follow-up note marks those two "Not as predicted" and revises the "Why". Natural ends help end-flat data a little and lose exact recovery of quadratics and cubics. A maintainer has to decide that trade. The probes at the earlier head 282df950 (pipeline 2992617933) gave the same split.

The flow also proposed the stricter test as !32. It opened !32 through the project's integration, so the merge request page shows HaoZeke as author; the commit 333f6beb is by the answer-check account. In pipeline 2992875937, !32's tests:tighten job (17492371813) passed. The new test passed on !32's head and failed on 282df950, as the check intends. It was red before because the job never installed bubblewrap, so the sandbox refused to start the tests on either commit. !33 (merge 3e5a59fc) adds the install. A ci-config test now fails if any sandboxed job skips it or sets the variable that turns confinement off. I moved the test's import below its env skip guard in commit ac73d758. The assertions are unchanged. On head 1ace122d (pipeline 2993073691) tests:tighten (job 17493701781) exits 0 on the head and 1 on 282df950. The flow's stricter-test notes on !3 quote a pass and a fail from commits that bench:bisect built, such as f8e7ce7a at 282df950. Those figures are not from !32's pipeline, and !32's job confirms them. !32 is not merged.

Lifecycle stages

Plan. Issue #2 tracked the method fix that landed in !20. The flow can open an issue when the fault is on the target branch.
Create. The flow writes the stricter test: commit 66ac80d in !7, commit 333f6beb in !32.
Verify. test, the bench:* jobs and forms:sollya ran in pipelines 2993042357 (!3) and 2991617555 (!20). tests:tighten passed in 2988373742 (!7) and 2993073691 (!32). probe:spec ran in 2993080545. On main, pipeline 2992686721 ran errprop and proofs:lean. The Lean proofs cover the Matérn 5/2 factor, the Lennard-Jones pair term and the standard rounding-error bound.
Secure. semgrep-sast and secret_detection run on merge requests, dependency-audit on tags. The probe job drops stored cloud credentials before it runs package code.
Package and Release. Tag pipeline 2988437089 ran build, sbom, publish:registry, release-notes and release for v2026.10.25.dev1, which predates the !20 fix. Its sci job failed with exit code 3; it is allowed to fail, so the pipeline is green.
Configure. ci-config tests the CI scripts, the flow and the agent config together, including a check that every sandboxed job installs bubblewrap. deploy/cloudrun/ has the Cloud Run deploy code.
Monitor. sci reads the CPU time of the jobs it can see and estimates a Software Carbon Intensity score. On main pipeline 2992686721 it read one job of six, ci-config, at 0.461 gCO2e.
Govern. guard:main checks who merged each commit on main, and Pages and the registry publish a person's merge and refuse an agent's. guard:mr fails a merge request pipeline that an agent account starts outside answer-check/*. The logs on record show these guards passing, not refusing.

How we built it

.gitlab-ci.yml is the pipeline and .gitlab/duo/flows/answer-check.yaml the flow. The ci/ scripts do the work: bench_gate.py, bisect_answers.py, sandbox.py, probe.py and report.py. The flow measures with the merge base's ci/ scripts, so the merge request under review cannot change how it is measured. integrations/asv/ records each pinned answer as an asv benchmark. proofs/ (Lean 4 with Mathlib) and studies/errprop/ back the kernel identities and the float32 error study. Main pipeline 2992974988 ran 893 tests of the CI scripts: 892 passed, 1 skipped.

GitLab pipelines also record each run of the flow. As of 08:42 UTC on 27 Oct 2026, GitLab lists 190 pipelines started by the flow. The flow's account (user 43724345) has posted 63 notes across 6 merge requests (!1 2, !2 14, !3 24, !17 13, !20 9, !21 1). The 24 on !3 include the 4 duplicates described under Challenges. Both counts work without a sign-in: GET /api/v4/projects/88035154/pipelines?source=duo_workflow (the x-total header), and /-/merge_requests/<iid>/discussions.json for every merge request, paged with X-Next-Page-Cursor, counting non-system notes by that user. The /notes API route needs a sign-in. Job logs are at /-/jobs/<id>/raw.

The ci/ scripts the flow uses are also packaged as a CI/CD component, templates/driftwatch.yml, with the package, test paths and tolerances as inputs. Since !34 this project's .gitlab-ci.yml includes it. The job here overrides the component's merge request rules: it runs on main, on HEAD~1 and HEAD, for spline_parabola alone. On main pipeline 2992974988 its driftwatch job (17493018358) passed: the sandbox engaged with bubblewrap, and spline_parabola came out the same bit for bit on 3e5a59fc and 11b8f9bd. That run had no code change to catch. On the demo branch demo/driftwatch-catch, the same two lines from !3 sit on top of a commit that lets the job run on that branch. Its driftwatch job (pipeline 2993327188) reports spline_parabola as a changed answer: 71 of 100 values moved, the largest by 0.00558 where 3.5e-08 is allowed. The job reports the change in its log and still passes; it does not fail the pipeline. The component is not published in the CI/CD Catalog yet.

Challenges

The flow posts after the pipeline passes, so a person can merge before the note arrives. !20 merged two minutes before its note. That note and its line thread arrived after the merge, and both are still unresolved. A pipeline-start trigger, which needs a Maintainer, would close that gap.
Each new head gets a new note and line thread. As of 08:42 UTC, !3 has 12 unresolved threads, a note and a line thread for each of the six heads the flow reviewed.
Before !34, bench:bisect on !3 named commit f8f29032 but not the line. That was a bug. The step diffed the commit against the merge base. After a merge of main, the merge base is not the commit's parent. !34 narrows each commit against its own parent and reports a root commit instead of stopping. Tests in ci/tests/test_bisect_answers.py cover both, and job 17493499556 on !3 shows the fix.
Bubblewrap cannot start inside the flow's own session, and the flow refused to run probes unconfined. !31 moved them into the probe:spec job, where they now run confined.
Two flow sessions started 4 s apart for 8e48a016. Both posted a verdict, a line thread and replies on !3. The second pushed the same test again onto the new stricter-test branch, opened as !37. I closed !37 as a duplicate of !32 and resolved the copied threads. The earlier session opened !37 itself. The flow opens a stricter-test merge request for each new head. It does this even when one for the same code is open. The claim fix does not stop that. Since !26 a run skips its note when the flow's account has already reported that head. Both sessions read the notes before either had written one. Both came from one pipeline, 2993042357. I counted the successful pipelines since 25 Oct that ended in a Pages deploy, as of 08:40 UTC on 27 Oct. At least 19 of 80 had started two or three sessions. The count rises to 25 of 80 if sessions that start up to 5 s before the pipeline finishes count too. None of 78 others had. !39 (merge 1ed77c47) merged a fix. Before a pipeline's run writes, it now claims the head by creating a branch. A second run on that head should find the branch taken and stop without posting. !39's tests check the prompt's wording. A claim that fails for any other reason lets the run go on. To test the stop, I opened !40, a draft that reapplies !3's two lines on main and is not for merging. Before its pipeline passed, I created the claim branch answer-check/claim-40-b0810f00 myself (commit e6f2bc3e), standing in for the run that wins. Pipeline 2993328150 then started two sessions, 13 s apart: 2993361483 and 2993362153. Both failed to create the claim, answered "nothing to check: another run has b0810f00" and stopped. !40 has no note from the flow, and no branch was pushed. Two flow runs racing for the claim are still unobserved. If a run dies after its claim, later pipeline runs on that head stay blocked until someone deletes the branch.

Accomplishments and what we learned

On !2 the flow told a float32 workaround apart from a method fix, with numbers, and refused to bless the new values. The fix landed in !20 and upstream in PR #52. On !3 its probes corrected two of its own predictions before a person had to.

Those two predictions carry the lesson. A green pipeline says the tests passed. For numerical code the reviewer also needs the old answer, the new answer and the exact answer side by side. An agent's explanation needs measuring too.

What's next

Watch two flow runs race for a claim that does not exist yet.
On a new head, reply in the flow's existing thread and reuse its open stricter-test merge request instead of opening new ones.
Publish the CI component in the CI/CD Catalog and run it on a second scientific Python project.

When the work was done

Submissions opened on 5 Oct 2026. The rgpycrumbs library predates the hackathon. The earliest Driftwatch measurements were local runs on 22 and 23 Oct, committed under ci/demo/results/. The site's "Two measurements" section shows one of those local runs next to the same input measured on GitLab. Its review pages for !2 and !3 are built from GitLab's records. The GitLab project was created on 24 Oct. Every measurement quoted above comes from its pipelines and the flow's notes, dated 26 and 27 Oct (UTC). Of !1 to !40, !3, !32 and the demo !40 are open, !25, !30 and !37 were closed, and the rest are merged.

Built With

bubblewrap
gitlab
gitlab-ci/cd
gitlab-duo-agent-platform
gitlab-pages
hypothesis
jax
numpy
python
scipy
sollya
sympy