Earshot

The idea

You walk into a loud party and someone across the room says your name. Everything else drops away and you hear them. Researchers call it the cocktail party effect. I wanted to turn that into a game.

In Earshot you sit in a room full of people talking at once. You cannot speak and you cannot walk up and interrupt. You can only listen. Cup a hand behind your ear, turn toward a conversation, and the noise around it falls away until you can hear every word.

How it plays

Someone has taken something. A retirement gift, a locket, an awards trophy. The host tells you what is missing and then the room starts talking. Every group has something to say, and some of it matters.

You catch clues by listening. Lift your hand toward a group and hold it there, and the muffled chatter opens up. Catch the clue, and a chime tells you so. Once you have heard enough, you name who did it and where the thing is, by pointing at an answer and holding.

The playable build has three cases, and each one adds a new rule:

1. The Retirement Party. A small room, three conversations. This is the teaching case.
2. The Golden Anniversary. A moonlit conservatory, four conversations, and a pianist with a secret.
3. The Lumen Awards. A night-time awards gala with a pool, a stage, a green room and six conversations. You have seven minutes. You walk between groups by pointing at glowing markers, keep a notebook of what you overhear, and look down at it to accuse.

Why it works with only your hands

The game needs no controller, ever. Everything is done with a hand or your head:

• Listening is a cupped hand near your ear. The closer it is and the more it points at a group, the clearer that group gets.
• Choosing an answer is pointing and holding for about a second. A bar fills so you can see it working.
• Walking (Case 3) is aiming at a marker and holding. The camera glides there, so you stay seated the whole time.
• Accusing is looking down at your notebook.

You can play the whole thing seated. If your hands are not tracking well, or you cannot raise them, looking straight at a group works as a slower fallback. Nothing in the game requires standing, reaching or fast movement.

The sound is the game

Each conversation is a separate audio source placed in the room. The audio is spatialised with HRTF, so you can tell where a group is without looking. Out of focus, a group is quiet and dull, like hearing through a wall. As you look at it and raise your hand, a filter opens and the volume rises, so the words come up out of the noise.

The guests' mouths move to the level of the voice, so you can see who is speaking. Wrong answers get a low buzz, right ones a bell, and solving a case brings a fanfare, applause and, at the gala, fireworks.

Every spoken line is captioned

The game is built on sound, and I did not want that to shut anyone out. Every line the host speaks, and every clue you catch, shows as a caption in a strip just under your eye line. Long speeches are cut into short pieces that follow the voice.

The conversations are captioned too. When you listen to a group, the words they are saying appear as they say them, so you can follow a case by reading as well as by hearing. In the gala, each clue also lands in the notebook as text. Captions are on by default, and adding ?nocaptions to the link hides them.

Case 3: where the clock starts

The gala is the part I am proudest of. The evidence is spread across six groups and runs on a timer. A line only counts if you are close enough and listening while it is spoken, so you have to choose where to spend your seven minutes. Some conversations are red herrings. Some groups have a clue in the first half and another near the end.

Every clue you catch lands in a notebook that follows your head. Get enough notes, look down, and you can accuse early, before the clock runs out. Your score depends on how many key clues you caught, how much time was left, and how many wrong guesses you made.

Reasons to come back

• Case 2 unlocks when you solve Case 1, and Case 3 when you solve Case 2.
• Guessing does not work. Two wrong answers and the trail goes cold, and you have to go back and listen again before you can try the quiz.
• Case 3 gives you a rank from "Lucky Guess" to "Perfect Ear" and saves your best score.
• Play on consecutive days and your streak shows in the case file.

How I built it

Earshot is built with the Immersive Web SDK (IWSDK), three.js and the Web Audio API. It runs from a link, with nothing to install.

• The rooms are built in code. The parlor, the conservatory and the whole gala (sky, pool, pavilion, string lights, LED wall that moves with the voices) are drawn with canvas textures, so there are almost no image files to load.
• Guests are painted figures with a closed-mouth and a talking pose, swapped by voice level.
• The audio graph gives each group its own panner and low-pass filter, driven by how directly you face it and how close your hand is.
• Walking is a smooth eased glide of the player rig with a soft vignette, steering around the pool and the stage.
• The captions come from my own voice recordings. I ran them through an open speech-to-text model on my laptop to get the words and their timings, then fixed the mistakes by hand.
• Progress, best scores and the streak are stored in the browser.

How the hands work

• The hands are read from the grip spaces and the ray spaces on the player rig each frame.
• "Cupping" is a score from 0 to 1. It combines how close a hand is to your head (full strength within about 30 cm, fading out by 50 cm) and how well the direction from your head to that hand lines up with the direction to a group. A score of 0.7 or more counts as listening.
• Answer cards are chosen by a pointing ray, or by your gaze, held for 1.2 seconds. A progress bar fills while you hold.
• In Case 3, glowing markers are chosen the same way, with a shorter hold, and the player rig glides to the spot.

Challenges

• Making "listening" readable without any button. The gold ripple at the hand, the orb that grows around a group, and the filter opening all had to say the same thing at once.
• Keeping people in a room looking like they are talking to each other, not to the camera. Pairs and small groups face a shared point and turn only a little.
• Keeping captions short and in time. A two-minute conversation loop has to show the right phrase at the right second, and a long speech has to be cut into pieces that follow the voice.
• Case 3 being a big scene that has to run in a browser on a headset. I kept everything as simple materials with no real-time lighting, and I have not been able to measure frame rate on a device yet.

What I learned

Sound can carry a whole game, if you let silence and muffled noise do the work. The best moment in the game is the half second when a group snaps into focus.

What's next

• More cases, each in a different room, with new voices and new rules for what makes a clue count.
• A daily case that is the same for everyone, so people can compare scores.
• A shared mode where a second player is in the room with you and you trade what you heard.
• Testing and tuning on a headset.

Built With

• github
• hand-tracking
• html5
• iwsdk
• javascript
• meta-quest
• spatial-audio
• three.js
• vercel
• vite
• web-audio-api
• webxr