Cue
Hear the game, see the game.
Inspiration
The idea started with one specific picture: a deaf or hard-of-hearing kid on a mainstream youth soccer team, standing on the field while their coach shouts instructions from the sideline — "Press!", "Cover Alex!", "Switch!" — and every single word sails past them. They're not slower, weaker, or less tactically aware than their teammates. They're just missing an entire channel of information that the rest of the team gets for free.
That's not a skill gap. It's an information gap. And information gaps are exactly the kind of problem software is good at closing.
We didn't want to build anything medical, and we didn't want to build anything that framed hearing loss as a problem to be fixed. Cue doesn't diagnose, treat, or "correct" anything. It just takes information that has always been audio-only and gives it a visual form — the same instruction, a different channel. That distinction shaped almost every design decision that followed.
What it does
Cue is two apps sharing one real-time session. A coach opens the Coach app, taps Start Session, and coaches exactly the way they always have — out loud, from the sideline. Behind the scenes, Cue transcribes that speech, runs it through a purpose-built interpretation engine, and — when it's confident — pushes an instant visual cue to a Child app running full-screen on a second device: a bold icon, an animated directional arrow, or a player's name and jersey number.
Thirteen tactical commands are supported out of the box (Press, Cover, Pass, Mark, Hold, Stop, Spread Out, Switch Play, the four Move directions, and Shoot), each with real synonyms a coach might actually say instead of the "canonical" word — "attack" resolves to Press, "protect" resolves to Cover, "push up" resolves to Move Forward. Player targeting understands both names ("cover Alex") and spoken jersey numbers ("mark number ten").
How we built it
The center of the project is the Sports Command Engine, a deterministic six-stage pipeline rather than a simple keyword match:
$$ \text{Speech Transcript} \rightarrow \text{Normalizer} \rightarrow \text{Tokenizer} \rightarrow \text{Player Resolver} \rightarrow \text{Intent Resolver} \rightarrow \text{Confidence Engine} \rightarrow \text{CommandEvent} $$
Each stage is a small, pure, independently testable class:
- TextNormalizer lowercases, strips punctuation, and collapses immediately repeated words, so "press press press!" — which real coaches actually say under pressure — still resolves to one clean command.
- Tokenizer splits the cleaned text into words and drops filler words that carry no tactical meaning ("the," "to," "let's").
- PlayerResolver scans the remaining tokens against a roster for a name, alias, or spoken jersey number, and strips that reference out before intent matching runs.
- IntentResolver scores every entry in a command dictionary against what's left, rewarding a match on the primary keyword more than a match on a synonym, and requiring a direction token for any "move" command before it can even be a candidate.
- ConfidenceEngine turns the ranked candidates into a single accept-or-reject decision.
The confidence score itself is a simple, fully deterministic formula — no model, no randomness, every term traceable back to a specific reason:
$$ C = \text{clamp}\left(\frac{|T_{\text{match}}|}{|T|} + b_{\text{primary}} + b_{\text{direction}} - p_{\text{ambiguity}} \pm b_{\text{player}},\ 0,\ 1\right) $$
where $|T_{\text{match}}|$ is the number of trigger words matched, $|T|$ is the total token count, $b_{\text{primary}}$ rewards matching the canonical keyword rather than only a synonym, $b_{\text{direction}}$ confirms a direction token for movement commands, $p_{\text{ambiguity}}$ penalizes the score when a second candidate is scoring nearly as high, and $b_{\text{player}}$ is a bonus or a steep penalty depending on whether a required player was actually resolved. If $C$ falls below the acceptance threshold, the command is rejected outright — never guessed, never shown with false confidence.
Every one of those terms gets written into a human-readable explanation list attached to the resulting CommandEvent, which the Coach app renders directly in an expandable "Why?" panel. The engine is never a black box, even in a hackathon build.
On the infrastructure side: Flutter with a feature-first architecture (coach, child, and shared UI features sitting on top of a shared core layer of engine, models, and services), speech_to_text for on-device continuous recognition, Firebase Realtime Database syncing accepted commands between the coach and child devices in real time, provider for state management, and a Material 3 theme built around a high-contrast palette chosen for outdoor sideline visibility and common forms of color blindness.
Challenges we ran into
Deciding when to say nothing. The easy version of this project always shows the closest-matching command. We didn't want that. An incomplete "Cover—!" with no name attached, or a genuinely ambiguous utterance sitting between two plausible commands, is worse to show confidently than to reject. Building the ambiguity penalty and the missing-player penalty into the Confidence Engine — and tuning the acceptance threshold so "cover" alone with no target actually gets rejected rather than squeaking through — took real iteration and a lot of test cases.
Coaches don't talk like a command-line interface. Real sideline coaching is repetitive, fragmented, and full of filler words and synonyms. The normalizer's word-repetition collapsing and the dictionary's synonym lists both exist because the naive "exact match" version of this engine failed constantly against realistic test transcripts.
Android's speech recognizer quietly gives up. It stops listening after a short pause, which would have silently broken a live coaching session mid-game with no obvious symptom beyond "commands stopped working." SpeechService watches for the recognizer's done status and automatically restarts listening while a session is still active, so a natural pause in coaching never ends the session.
Accomplishments that we're proud of
An interpretation engine that's genuinely explainable end to end — every accept or reject decision the coach sees comes with the exact reasoning behind it, not a confidence percentage with no context — paired with a child-facing screen that stays calm between cues and doesn't overwhelm with information, which matters just as much as recognizing the command correctly in the first place.
What we learned
Accessibility tools live or die on trust, not on maximum recall. A system that occasionally shows the wrong cue teaches a kid to stop trusting the device entirely, which is worse than a system that sometimes shows no cue and stays quiet. That reframing — optimize for confident correctness, not for always having an answer — ended up shaping the entire engine, from the ambiguity penalty down to the decision to reject rather than default.
What's next for Cue
A coach-configurable roster and vocabulary so any team can onboard without a code change, session codes to support multiple coach/child pairs running at once, real authentication and locked-down database rules, haptic cues for heads-down moments, and eventually a wearable output so the cue doesn't require pulling out a phone mid-play.
Log in or sign up for Devpost to join the conversation.