-
-
Thumbnail
Inspiration
Sovereign AI is only useful for software work if we can determine whether the code it produces actually works — without trusting the model's own claims and without exposing private code or hidden evaluation data.
AirPatch was built as a reproducible evaluation and code-repair pipeline for Apertus.
What it does
AirPatch gives Apertus a programming task, faulty source code and visible examples. Apertus proposes a corrected program.
The proposal is then evaluated by a deterministic pipeline that does not use an LLM for judging. Each candidate is executed in isolated, offline Docker containers against hidden test cases.
The pipeline records build results, passed tests, runtime, reproducible error fingerprints, model calls and token usage.
AirPatch also supports verifier-guided repair rounds without revealing hidden test data to the model.
How we built it
The pipeline consists of seven stages:
- propose
- apply
- plan
- execute
- verify
- feedback
- report
Apertus is used only for generating code. Correctness is decided by deterministic execution.
We evaluated both swiss-ai/Apertus-v1.5-8B and swiss-ai/Apertus-v1.5-70B through the CSCS inference endpoint.
All model answers were recorded so the complete evaluation can be replayed offline without an API key.
Validation and results
Before evaluating the models, we validated the measuring instrument itself.
- 51/51 reference solutions pass
- 51/51 unchanged faulty programs are rejected
- 102/102 near-miss variants are rejected
- isolation and leak probes were tested with a positive control
On 40 previously unseen test tasks:
- Apertus 8B: 2/40 solved with one answer
- Apertus 70B: 2/40 solved with one answer
- Both models: 3/40 solved when allowing any of three sampled answers
- Verifier-guided repair rounds added no solved tasks
This is deliberately reported as a weak negative model result rather than turned into a positive claim. The contribution of AirPatch is the ability to measure such results reproducibly.
Reproducibility
Every model answer used in the experiment is included in the repository.
The evaluation can therefore be replayed offline without an API key:
cd track_2b && make run
Replay runs are explicitly labelled REPLAY and do not create new Apertus results.
Challenges
The hardest part was not generating code — it was building an evaluation system whose own verifier, isolation, determinism and recovery behaviour could be tested independently.
A post-evaluation audit found a resume-related defect. We reproduced it, verified that it did not affect the frozen experimental results, fixed it without changing those results, and added regression tests.
What we learned
Model evaluation infrastructure deserves the same scrutiny as the model being evaluated.
A reproducible negative result can be more useful than an impressive but unverifiable benchmark number.
Limitations
The benchmark contains small Python tasks and does not represent large production repositories. The 40-task test set also gives relatively wide statistical intervals.
AirPatch is therefore an evaluation instrument and reproducibility framework, not evidence that these results generalize to all software-engineering tasks.
Log in or sign up for Devpost to join the conversation.