-
-
Rieux: which machines does this whisper.cpp release still run on? Every number comes from a script in the repo.
-
The Rieux page: every x86-64 machine back to x86-64-v1 can still run this release.
-
The build machine sets the floor. In 11 of 38 commits, upstream's main and main-cuda images from the same source have different CPU floors.
-
The backtest of 121 published images, and the CPU proof under QEMU: only the all-variants build runs on all four old CPU models.
-
The repair loop on GitLab: 17 fix MRs from the agent flow, median 1.8 minutes after the failed pipeline. A person decides the merge.
-
Languages are a floor too. On 50 FLEURS sentences per language, English and Spanish are served at every model size. Six languages are not.
-
The cost of inclusion: the all-variants build uses 0.4% less CPU time than the folk fix, inside the 5.1% noise. No measurable cost.
-
Footprint (SCI, Impact Framework): 0.00196 g CO2e per audio-minute in northamerica-northeast1.
-
ΔM: the devices that a floor raise could throw away.
-
Fix MR !18, opened by the floor-repair flow after the tag pipeline of v1.9.5-rieux.1 failed. Its gate held every floor.
Inspiration
whisper.cpp is a speech-to-text engine made to run offline on ordinary computers. Yet its official Docker images crash with “illegal instruction” on perfectly normal machines.
In issue #3510, the image main-a1867e0 dies at startup with exit code 132, on a 2024 laptop chip and a 2020 desktop chip. A stale bot closed it without a fix, and a user reported it again on 2026-10-03. In issue #3814, main-cuda crashes on a 2020 desktop CPU. A stale bot closed that one too. These are not one-offs. Searching whisper.cpp for “illegal instruction” turns up 39 issues and pull requests. We read a seeded sample of 20: 13 were crashes from instructions the CPU could not run.
We traced it to the build, not the code. All 7 upstream Dockerfiles use ggml’s default, GGML_NATIVE=ON, so each image is compiled for whatever CPU the CI machine happened to have. Ordinary CI can never catch this, because it tests on the machine that built the image: the one CPU guaranteed to pass. The popular fix, GGML_NATIVE=OFF, is a trap: the binary still needs a CPU from about 2013 or later. And upstream already has the right recipe (GGML_CPU_ALL_VARIANTS) in its release workflow. It just never reached the Docker workflow, and nothing checks that the two agree.
Every release raises the ceiling. We built the thing that watches the floor.
What it does
Before and after Rieux:
- Who can use a release. Before: nobody knows; the build machine decides. After: a Floor Manifest for every release, with the CPU, memory and connectivity floors, each with proof.
- A floor rises. Before: users find out when the image crashes, and a stale bot closes the issue. After: the pipeline fails, a flow opens a fix MR within minutes, and the gate checks the fix again.
- The fix doesn't hold. Before: no process. After: a script applies upstream's own inclusive config. If that fails too, Rieux drafts a Leave-Behind Notice, and only a human can approve it.
- Release notes. Before: a list of code changes. After: "Who this release serves", written from the signed manifest.
- A user reports a crash. Before: wait for a maintainer. After: a triage flow reads each new issue and points a floor report to the matching tested CPU and the latest release. Tested so far: it correctly skips an issue that isn't about a floor (issue #2).
On every push, merge request and tag, the GitLab pipeline:
- Builds the image with upstream's own recipe.
- Scans every ELF file for the highest x86-64 level in code that always loads: the static CPU floor.
- Runs
whisper-cliunder QEMU on four CPU models (qemu64, Nehalem, SandyBridge, EPYC-Milan), with a positive and a negative control on each. - Squeezes it: a transcription under cgroup memory limits, and one with no network at all, under strace.
- Judges it. The Judge is a script, not a model. Each floor comes back held, lowered, raised or inconclusive.
When a floor rises, the floor-repair flow on the GitLab Duo Agent Platform takes over. The agent reads the scan and the build files and opens a fix MR. It may touch only the build overrides, the upstream Dockerfiles and deploy/; the policy, the tools and the pipeline belong to code owners. The fix goes through the exact same gate. When the Judge passes, attest signs the Floor Manifest as an in-toto attestation (keyless, with the GitLab ID token), and herald writes "Who this release serves".
Agents propose. The Judge disposes. Humans exclude.
How we built it
- One Python package,
floor: 36 modules and 27 test files. 280 tests passed in thecheckjob of pipeline 2973711384. - One GitLab pipeline: check, build, service, scan, probe ×4, memory, offline, judge, fallback, notice-draft, attest, herald, release and pages. The live service on Cloud Run (northamerica-northeast1) runs the image that the gate held for
v1.9.5-rieux.1-floor.1, deployed withinfra/quick-deploy.sh.infra/main.tf(OpenTofu) and two pipeline jobs are ready to deploy every signed release automatically, with Workload Identity Federation so CI never holds a key file. - Four flows in
.floor/flows/, saved as private AI Catalog flows: floor repair (on a failed pipeline), field triage (on a new work item), release summary (on a passed pipeline) and a trigger probe. - The Rieux page on GitLab Pages: the backtest timeline, the floors, the languages, SCI, ΔM and the repair loop, all from committed evidence.
What we added after Oct 5
The GitLab project is whisper.cpp at upstream commit 0c7be696 (2026-10-12, MIT, full history) plus the Rieux files. We didn't change a single whisper.cpp file; we only moved its README to README.whisper.cpp.md. Every Rieux file is new, written from 2026-10-14 on:
.floor/: thefloorpackage, its tests, the policy, the evidence, the flows, the notices, the scenarios and the build overrides..gitlab-ci.ymland.gitlab/: the pipeline and the code owners.deploy/andinfra/: the service image, the Cloud Run setup and the live deploy script.README.md: the Rieux README.
The git history shows exactly where upstream ends: the first Rieux commit on GitLab is 36521198 (2026-10-17).
Challenges we ran into
The build machine kept moving under us. 20 of our 23 main builds of upstream’s recipe on GitLab landed on a runner with AVX-512; 3 did not. The same source gave x86-64-v4 or x86-64-v3 on GitLab, and x86-64-v3 on a GitHub runner. That is exactly why the gate checks every build, not just the recipe. Emulating real images. QEMU runs the image’s own loader and libraries, so the probe needs the image’s file system as a sysroot, with every absolute symbolic link rewritten as a relative one. Slow emulation. The EPYC-Milan probe ran past its 2400-second limit in the first gates. We shortened the audio context but kept the full 11-second sample. Since then the job takes 1156 to 1464 seconds on the Intel runners (11 MR pipelines) and 223 to 353 seconds on an AMD EPYC 7B13 runner (MRs !15 and !16). Trusting the agent’s own definition. The AI Catalog doesn’t check the tool names in a flow. So after every save we read the stored definition back and compare its SHA-256 with the file in .floor/flows/
Accomplishments that we're proud of
The loop runs on real pipelines.
We tagged v1.9.5-rieux.1, and its gate failed (pipeline 2987108367). 1.8 minutes later the flow had opened fix MR !18. Its gate held every floor (pipeline 2987119941), we merged it, and the main pipeline of the merge (2987186947) signed the manifest, wrote the release notes and cut v1.9.5-rieux.1-floor.1. After the tag, no human wrote a line of the fix; the only human step was the merge. The flow answered every failed pipeline: by 2026-10-25 it had opened 17 fix MRs (!1 to !12 and !14 to !18) for 17 failed pipelines. 15 arrived 1.6 to 2.9 minutes after the failure (median 1.8); the other two took 4.3 and 8.9 minutes. Source: the GitLab API (.floor/evidence/repair/repairs.json); the Rieux page shows the table. The gates of fix MRs !3, !5, !6, !7, !10, !11, !12, !14, !15, !16, !17 and !18 held every floor. For MRs !15 and !16 (pipelines 2970973362 and 2973807759), every single job passed: four QEMU probes, memory, offline, the Judge, attest and herald. The attest job signed and verified the manifest (pipeline 2961110649, job 17270217868). The human veto works. MR !13 proposed removing ggml’s baseline CPU variant (about 1 MB) from the image. The gate raised the CPU floor to x86-64-v2 (on qemu64 the image stopped), the fallback had nothing to offer, and notice-draft wrote a Leave-Behind Notice: 11 to 12 of every 10,000 x86 users affected, ΔM 0.7 to 35 kg CO2e. We refused it and closed the MR. The Rieux page records the decision. The first main pipelines are red on purpose. Upstream’s recipe takes the CPU floor of whichever GitLab runner built it (x86-64-v4 on an Intel Xeon Platinum 8581C, x86-64-v3 on an AMD EPYC 7B13), and our policy asks for x86-64-v1. Honest scope: two of the three loops we planned ran, the default-flags repair (the release above) and the notice (refused, so no release). The folk-fix repair did not run.
Environmental impact (SCI).
0.00196 g CO2e per audio-minute (SCI, ISO/IEC 21031, Impact Framework 1.1.1) for the all-variants build in Google Cloud northamerica-northeast1. Embodied carbon is 84.6% of that. The same work in us-central1 comes to 0.0131 g, 6.7 times more. One release check by Rieux itself costs 0.36 g CO2e on the GitLab runners: 486 vCPU-seconds, metered in each job’s cgroup. That is a lower bound. The bigger lever is hardware. When a release raises the floor, some people must replace their device. ΔM estimates the embodied carbon of those devices. It is a scenario with a range, and we never add it into SCI. Per 10,000 x86 users: raising the floor from x86-64-v1 to x86-64-v2 (a build without the baseline CPU variant) gives 0.7 to 35 kg CO2e; to x86-64-v3 (the folk fix), 30 to 1,651 kg CO2e; to x86-64-v4 (upstream’s recipe on an AVX-512 build machine), 889 to 87,338 kg CO2e. Inputs: the Steam Hardware Survey (September 2026), two Dell product carbon footprints, and an assumption that 1% to 5% of affected people replace their device. The scenario files are in .floor/scenarios/, and the Rieux page shows them.
What we learned
A CPU floor is a property of the build machine, not just of the code. An agent earns trust when a script checks its work. Our Judge is a script, and every fix MR goes through the full gate again. Raising a floor isn’t always wrong. But a human should make that call, and the release notes should say plainly who gets left behind.
What's next for Rieux
Bring the gate to llama.cpp and the other ggml projects, which share the same build options. Add more floors: GPU drivers, ARM levels and the operating system version.
Log in or sign up for Devpost to join the conversation.