PitchSide

Inspiration

Pitchside started with a real person: a mom who began watching the World Cup because Canada qualified, and had no idea why the referee kept blowing the whistle. Millions of first-time viewers are joining this tournament , hosted across Canada, the US, and Mexico, and everything built for football fans assumes you already are one. Wikipedia is too dry, Reddit is too deep, and the broadcast moves on before you can ask "why was that offside?" We wanted the thing she actually needed: a coach in her pocket that explains the game at her level, in her language, out loud.

What it does

Pitchside is a voice-enabled AI World Cup companion. You pick one of three coaches — each with their own personality and their own ElevenLabs voice — and ask anything by text, voice, or by pointing your camera at the TV. A beginner gets a warm plain-language answer from El Pocho; a FIFA player gets banter from The Special One; a tactics nerd gets xG and line-height analysis from El Maestro. Live questions ("who plays in the next semifinal?") are grounded in real Google Search data with visible citations. Formation questions come back with an animated tactical whiteboard drawn by Gemini. And "Coach Call" turns it into a true hands-free conversation: speak, get a spoken answer, speak again — no touches in between. It works natively in English, French, and Spanish.

How we built it

Two APIs, one clean division of labor: Gemini writes, ElevenLabs performs. A React frontend talks to a FastAPI backend; Gemini Flash handles chat with persona and language as system-prompt parameters, Google Search grounding exposed as a tool the model invokes autonomously (no keyword routing — the model is the router), and structured output for the tactical diagrams. ElevenLabs handles text-to-speech with intensity-mapped voice settings, Scribe speech-to-text for voice input, and per-line "dramatic performances" where hype scripts are voiced line-by-line at different intensities and stitched into one MP3. Audio is cached by content hash so rehearsed lines play instantly. The hands-free Coach Call runs on a WebAudio voice-activity detector: 1.5 seconds of silence ends your turn, and the mic reopens automatically when the coach finishes speaking. Deployed on Vercel + Render so anyone can scan a QR code and try it.

Challenges we ran into

The best one: combining Gemini's structured JSON output with Search grounding silently drops the citation metadata. Our sources row — the proof that answers aren't hallucinated — kept coming back empty. We rebuilt the response format around a parsed tag line instead of JSON mode, which ironically made the code simpler. ElevenLabs taught us that quota exhaustion arrives disguised as a 401 (we burned the free tier mid-hackathon and spent an hour suspecting our own auth code), and that Scribe cheerfully transcribes background noise as "(percussive sound effects)" — which our chat then answered as a question. Windows added a flat two-second tax on every request because localhost resolves IPv6-first. And hanging up a Coach Call while audio was still being generated created a race where the coach started talking after you hung up — a proper distributed-systems bug hiding in a hackathon UI.

Accomplishments that we're proud of

We shipped the entire PRD — and then a second one. All ten planned features (personas, trilingual chat, image understanding, search grounding, commentator TTS, multilingual audio, voice input, session memory, accessibility, hype generator) made it, then we layered on a standout pass: three coaches with three distinct ElevenLabs voices, hands-free Coach Calls, Gemini-drawn tactical whiteboards, camera-to-explanation, and a live match-day ticker. A real hands-free conversation, not a demo trick. Our voice-activity detection ends your turn after a natural pause, the answer speaks itself, and the mic reopens — we verified three-turn conversations with zero touches in between. Answers you can check. Live-data questions show their sources right in the chat bubble. During testing, the app correctly reported semifinal fixtures — teams, dates, and venue cities — hours after they were decided. The same question genuinely gets three different answers. Not simpler wording — different structures: an analogy from El Pocho, a tactical note with swagger from The Special One, line-height and xG analysis from El Maestro, each in their own voice. It's live on judges' phones. Deployed on Vercel + Render behind a QR code, tested end-to-end on iPhone viewports, with cached demo audio replaying in 28 milliseconds. We sweated the details: answers in French speak fluent French, reduced-motion users get a still background, screen readers announce replies, error messages localize, and a hype script audibly builds from calm to explosive inside a single MP3

What we learned

Prompts are a real engineering surface: persona, language, grounding discipline, and output format all live in one system prompt, and small wording changes are the difference between three personas that sound different and three that just use different vocabulary. We learned to treat APIs' failure modes as features to design for — caching, retries, graceful degradation everywhere — and that the most impressive demos come from composing capabilities (grounding + TTS + VAD = a phone call with a coach) rather than adding more of them.

What's next for PitchSide

Live match mode. A second-screen companion that follows the game in real time — grounded minute-by-minute updates, with the coach jumping in after big moments ("here's why VAR is checking that")