Inspiration
Apertus 1.5 is a Swiss model with new audio input, and its published safety scores are text-only. We tested whether a request it refuses in writing gets real help when the same request is spoken.
What it does
One issue: spoken requests receive harmful help that the same text refuses.
On 60 StrongREJECT items and two seeds, both judges must mark an answer on-target harmful, and a hand audit has to agree. Echoes and mishearings are excluded.
- 8B: text refused 120/120. Confirmed harmful speech 43/360 (11.9%). Clear cases only 21/360 (5.8%).
- 70B: text refused 120/120. Confirmed harmful speech 20/353 (5.7%). Clear cases only 9/353 (2.5%).
Harmless spoken prompts are still answered, so this is not a blanket speech failure. Thinking-mode leaks, planted personal data, images, and several culture, bias, copyright, and factual probes did not meet their bars and are not submitted as issues.
How we built it
Python standard library, the Public AI endpoint, and Windows text-to-speech. Judges were Qwen-SEA-LION 32B and Gemma-SEA-LION 27B. make run replays the committed finding in Docker and does not call the model.
Challenges
The gateway cache could return the wrong condition. Synthetic voices are not human recordings. Raw harmful answers are kept out of the public repo. The finding file, audit, and report are in the GitHub project.
Log in or sign up for Devpost to join the conversation.