thumbnail
shush
ML-classifier inference

DEAD SILENCE

Inspiration

We started with a simple frustration: sometimes you want to send someone a voice message, but speaking out loud just isn't an option. On a plane, in a quiet library, or in a sauna, even a short message can feel disruptive. That made us wonder: could we communicate with almost no noticeable vocal or facial activity?

Breathing felt like an unlikely place to start. We already inhale and exhale; perhaps a microphone close to the mouth could recognize intentional patterns in those sounds and turn them into text. The idea was not to transcribe whispered speech, but to create a small language input out of breathing itself.

We could have built a perfectly normal keypad for our game. Instead, we deliberately overengineered that simple text input into a breathing-pattern recognition system. The challenge became part of the experience: communicating quietly is no longer just a story constraint—it is something the player has to do.

For the hackathon's space theme, we built a first-person spaceship game in Godot around that idea. The atmosphere was inspired by the unsettling feeling of being trapped and unable to communicate in Harlan Ellison's I Have No Mouth, and I Must Scream.

What it does

An unknown, hostile entity has wiped out the crew of a spaceship. One crew member remains. To escape, the player must move through the ship, complete quests, and solve riddles before time runs out. Make too much noise and the entity can detect and catch you; run out of time before finishing the quests and the game ends.

Some interactions ask the player to enter text. Instead of typing on an ordinary keypad, the player uses a boom microphone and deliberate breathing. The system classifies each moment as an inhale, an exhale, or no intentional breath. An inhale represents 1, and an exhale represents 2. Five digits form a letter: for example, 11111 is A and 11112 is B. A reference keyboard stays visible while the player is using breath input.

Pauses provide additional controls. After a completed letter, a short detected pause lets the player begin the next one, a longer pause adds a word space, and a longer pause sends the message. A separate five-breath code deletes the last letter. These controls turn a basic text field into an interaction that fits the game's central rule: stay quiet.

How we built it

We built the game in Godot and developed the breath-recognition pipeline in Python. The model takes microphone audio at 48 kHz, divides it into 50-millisecond frames, and extracts energy from six frequency bands. It summarizes the latest one second of those features and feeds the result to a scikit-learn Random Forest classifier.

The window slides forward every 50 milliseconds, so the system can update its prediction frequently while still giving the model a full second of context. We also require three consecutive matching predictions before changing the displayed state. A decoder then turns transitions between breathing states into digits, letters, spaces, deletes, and send events.

We packaged the model, the mapping, the decoder, the training and evaluation data, and a Python inference bridge for the game. The bridge sends structured events over a local connection so the game can display the current prediction, partial code, and entered text. We used MLflow to record experiments and latency measurements as we tested different approaches.

Challenges we ran into

Breath sounds are subtle, and real rooms are not silent. Fan noise, background conversation, microphone position, and changes in breathing intensity can all affect what the model hears. We collected examples of inhaling, exhaling, and no intentional breath—including recordings with background noise—so that the system had a third option besides forcing every sound into inhale or exhale.

A strong clip score did not automatically mean a good live experience. Our earliest two-class models achieved very high scores on short, separately recorded test clips, but continuous input introduced pauses, transitions, and background sounds that those tests did not capture. We tried a 50-millisecond classifier and later explored 1.8-second, 1.5-second, and one-second windows. Longer windows offered more context, but could also mix two breaths or delay recognition of a pause. The one-second version became our final model for the handoff.

Silence is a timing problem as well as a sound. A pause between breaths, the gap after a letter, and the pause used to send a message can all sound similar. We had to decide how the decoder should interpret detected silence and make those thresholds adjustable. The rolling audio window also means that a pause may be detected later than it physically begins.

We learned that preprocessing can make things worse. We experimented with a separate attenuation-based breath gate to reduce background noise. In live testing it also made quiet breaths harder to detect, so we kept the original path available and moved away from using the gate by default. We also explored marking breath regions in recordings; identifying exact boundaries proved much harder than assigning a label to an entire clip.

What we learned

Building DEAD SILENCE taught us to separate a promising model result from a reliable interaction. Our one-second model classified 20 of 20 original held-out breath clips and 21 of 22 newer held-out clips correctly in an offline check. Those results are encouraging, but the dataset is small, the clip labels are imperfect, and some recordings were collected close together. They do not establish continuous typing accuracy or performance for a new player in a different environment.

We also learned to treat the experience around the model as part of the engineering. A player needs to see the keyboard, the current partial code, and what the system thinks it heard. A fast prediction is useful only if the full sequence—from breathing, to classification, to a letter appearing—feels understandable and controllable.

What's next

We want to collect more data across people, rooms, microphone positions, and breathing styles, especially continuous sequences with known intended inputs. That would let us measure missed breaths, false activations, letter accuracy, and end-to-end delay instead of relying only on isolated clip scores. We also want to refine the pause timings and make the experience more comfortable as the game and input method evolve.

DEAD SILENCE began as an intentionally excessive answer to a simple keypad. It became an experiment in how a game can make a technical constraint feel like part of its world—and in whether an almost invisible action can become a way to communicate.