Clutch

Baseball Highlights Reimagined

Comment 1

Inspiration

As someone who isn’t enthusiastic about sports, participating in this hackathon was as challenging as I expected. While watching multiple baseball games to understand the sport better, I realized that baseball has its own terminology, keywords, and abbreviations that contribute to understanding the game. I often found myself searching for the meaning of phrases like "innings," "home run," "strikeout per 9 innings," and "1st-3rd base." That’s when the idea struck—what if watching baseball wasn't this complicated? What if the experience was tailored for both die-hard fans and newcomers?

That was when Clutch was born.

What It Does

Clutch is a progressive web app (PWA) that brings the best moments from every MLB game in a format that feels intuitive and engaging. Think of it as TikTok but for baseball fans. Users can swipe through personalized highlights, get instant explanations of gameplay moments, and learn more about baseball through interactive features. Whether you're a seasoned fan or new to the game, Clutch makes baseball more accessible and fun.

How I Built It

Backend Engineering

I built Clutch’s backend using Node.js and TypeScript, with Hono.js as the framework. The backend serves as the core engine handling video processing, AI-powered insights, and user preferences. I used PostgreSQL with Prisma ORM for structured data storage, ensuring efficient retrieval of game-related metadata, transcripts, and user preferences. Redis is leveraged for caching and as a job queue system to manage background tasks like video processing and highlight generation.

One of the biggest technical challenges was processing full-length MLB videos and identifying the best gameplay moments. To solve this:

  • Video Processing: I used FFmpeg to extract audio from game footage. This extracted audio is then fed into Google Cloud Speech-to-Text API, which generates a transcript of in-game commentary.

  • AI-Powered Highlight Detection: The transcript is analyzed using Gemini (2.0-Flash) to identify significant moments in the game. The model detects phrases like "Home run!", "Strikeout!", or "Unbelievable catch!" and extracts timestamps for these key events.

  • Multi-Language Support: The transcript is processed through Google Cloud Translation API, allowing non-English speakers to access translated versions of the commentary.

Once the metadata for highlight moments is generated, Inngest (a job scheduling system) handles asynchronous processing tasks, including video slicing and highlight playback generation.

Recommendation System

To personalize the highlight feed, I implemented a recommendation system based on the teams selected during onboarding. I used a Fisher-Yates shuffle algorithm to randomize and prioritize video playback for users. This ensures a mix of relevant and engaging content tailored to each user’s preferences.

AI-Powered Assistant

To make Clutch more interactive, I implemented an AI-driven assistant using Gemini 2.0-Flash and Exa.ai (a powerful AI search engine). This assistant provides real-time explanations and context for gameplay moments, allowing users to ask questions like "Why was this play important?" or "What’s the strategy behind this move?" The assistant references structured data from my database and external sources to deliver accurate insights.

Frontend Engineering

The frontend of Clutch is built with SvelteKit, ensuring a fast and lightweight user experience. TailwindCSS is used for styling, enabling a responsive design that adapts to different screen sizes seamlessly. The app follows a PWA architecture, allowing users to access it across devices with near-native performance.

For authentication, I used OAuth2, allowing users to sign in securely and sync their preferences across multiple devices.

Challenges I Ran Into

Coming Up With Ideas

One of the challenges I faced was coming up with an idea for baseball fans, considering I wasn’t a sports enthusiast. My first idea, "SluggerSidekick," was a virtual baseball mascot that provided live game insights. However, implementing animations in Rive proved to be more complex than expected, leading me to pivot toward Clutch.

Prompt Optimization

Working with AI models wasn’t as straightforward as expected. It was easier to explain baseball to a five-year-old than to get an AI model to consistently generate relevant insights. I had to fine-tune and optimize multiple prompts to ensure accurate gameplay analysis.

Video Processing

Processing MLB videos efficiently required testing eight different techniques before settling on the most effective one. From trying to detect gameplay moments using image grids and scene detection to experimenting with audio analysis, each approach had its limitations. The final approach—leveraging FFmpeg for audio extraction, Gemini for transcript analysis, and Google Cloud APIs for transcription and translation—proved to be the most reliable.

Time Constraints

Balancing Clutch’s development with ongoing exams and a full-time job made time management crucial. Despite these challenges, I was able to build and launch Clutch in just three weeks.

Accomplishments That We’re Proud Of

  • Developing Clutch from start to finish in three weeks, despite time constraints.
  • Successfully implementing an AI-powered highlight detection system that extracts key moments from MLB games.
  • Creating a personalized recommendation system that enhances the user experience.
  • Enabling multi-language support, making baseball highlights accessible to a global audience.

What I Learned

  • Building AI-powered video analysis systems requires multiple iterations and optimizations.
  • Handling large-scale video processing efficiently involves combining tools like FFmpeg, Google Cloud APIs, and AI models.
  • User experience matters—making baseball content more engaging requires more than just AI; it needs a seamless, intuitive interface.

What’s Next for Clutch

  • Live Game Insights: Integrating real-time game tracking so users can follow live baseball games with AI-generated summaries.
  • More Sports: Expanding Clutch to cover other sports like basketball and soccer.
  • Enhanced Personalization: Improving the recommendation algorithm to deliver even more tailored content.
  • Mobile App: Developing a native mobile app version for better performance and offline viewing capabilities.

Clutch is just getting started, and the journey ahead is exciting!

Built With

Share this project:

Updates