Inspiration
Apertus is unusually open for a modern language model. Its weights, training stages, serving stack, tokenizer, and evaluation tooling are all available for inspection. We wanted to use that openness for more than benchmark scoring.
Our question was simple: when Apertus gives the right answer in one representation, does it preserve that meaning when the same information moves through a different prompt, tool interface, language, or serving path?
That became APERTUS FRACTURE.
What it does
APERTUS FRACTURE is a reliability evaluation framework for Apertus 1.5.
It builds controlled pairs of inputs that should behave consistently, then measures where that consistency breaks. Instead of looking for isolated bad prompts, it searches for repeatable failure patterns with objective answers and matched controls.
The framework tests areas such as:
- tool-call and tool-result handling
- structured tool schemas
- numerical and multilingual representations
- semantic rephrasings
- multi-turn state
- differences across Apertus training checkpoints
Promising failures are retested on the full, unquantized model with fresh holdout cases before they are treated as real findings.
How we built it
We built a Python evaluation harness around Apertus 1.5 and its released checkpoints.
The workflow is:
- Generate synthetic test cases with known answers.
- Apply one controlled transformation.
- Run both versions through Apertus.
- Grade them with deterministic oracles instead of another language model.
- Reject weak or ambiguous effects.
- Reproduce promising cases on the full-precision model.
- Run fresh holdouts, statistical checks, and realistic transfer tests.
- Package validated findings into automated replays.
We also inspected the public Apertus serving and formatting code to understand whether some failures came from the model itself or from information being lost between the API, chat template, parser, and model.
The project includes tests, frozen experiment protocols, result ledgers, Docker support, and a make run reproduction path.
Challenges we ran into
The biggest challenge was avoiding false discoveries.
Several effects looked dramatic in 4-bit screening and disappeared on the full-precision model. We also found cases where an apparent model failure was actually caused by grading, formatting, unsupported input, or a weak control.
Another challenge was novelty. Many interesting LLM failures, such as sycophancy, multilingual weaknesses, schema loss, and tool-result confusion, already have prior art. We had to separate "new for Apertus" from "genuinely new failure pathway."
Compute was also a constraint. Apertus 8B in the reference configuration does not fully fit in our 16 GB GPU, so full validation used CPU offload and careful scheduling.
Accomplishments that we're proud of
We built a test system that is deliberately difficult to fool.
Every serious candidate is checked against fresh cases, matched controls, raw model outputs, model revisions, inference settings, and deterministic grading.
We also identified reproducible failures in real Apertus interaction paths, including cases involving parallel tool results, tool-schema information loss, user contradiction, multilingual number interpretation, and semantic comparison framing.
Just as important, we rejected many attractive findings when the evidence did not hold up. Quantization artifacts, weak controls, duplicate mechanisms, and known issues were kept out of the final claims.
The result is not just a list of model mistakes. It is a reusable framework for testing whether semantic information survives across Apertus interfaces.
What we learned
The most important lesson was that model reliability is often an invariance problem.
A model can know the correct answer and still fail when the same information is expressed through a different interface, language, tool result, or prompt structure.
We also learned that the serving stack matters. Some failures are not purely reasoning failures. Information can be dropped or changed before the model sees it, or while structured output is converted back into an API response.
Finally, open models make much deeper debugging possible. Because Apertus exposes its checkpoints and implementation, we could investigate not only whether something failed, but where the failure entered the pipeline.
What's next for APERTUS FRACTURE
The next step is to turn APERTUS FRACTURE into a reusable regression suite for future Apertus releases.
We want to:
- run the same tests automatically against new checkpoints
- expand coverage of tool and structured-output interfaces
- add more multilingual semantic invariants
- compare behavior across model sizes
- make checkpoint regression analysis automatic
- contribute minimal reproducible cases and suggested fixes upstream
The long-term goal is simple: when Apertus changes, APERTUS FRACTURE should make it easy to see not only whether benchmark scores improved, but whether important behavior quietly broke somewhere else.
Log in or sign up for Devpost to join the conversation.