Fan Frames - Personalized Fan Highlights

Fan Frames Banner
Fan Frames UI 1
Fan Frames UI 2
Fan Frames Architecture Diagram
Padres home win against the Dodgers, "cartoonish 3D model"
Yankees home win against the Diamondbacks, "oil painting"
Cubs home win against the Orioles, "cartoonish 3D model"
Nationals away win against the Cardinals, "oil painting"
Reds away win against the Giants, "cartoonish 3D model"

Summary

Fan Frames is an innovative application that enhances the baseball fan experience by generating personalized highlights through text, audio, and AI-generated images. The application addresses the challenge of information overload in sports media by providing targeted, engaging content based on specific teams, players, and time periods.
What sets Fan Frames apart is its creative use of team mascots for visual representation, ensuring kid engagement as well as both compliance with AI image generation guidelines and maintaining strong team identity connections.
The application is built on a streamlined architecture using Google Cloud services, including App Engine and Cloud Run Functions, while leveraging cutting-edge AI technologies such as Gemini 2.0 and Imagen 3 who are the stars of the show because previous models weren't that sharp. The platform integrates seamlessly with MLB's API to fetch relevant data, which is then transformed into personalized content through various AI models for text generation, translation, text-to-speech, and image creation. Despite some limitations with API access and AI model restrictions, Fan Frames demonstrates a practical solution for enhancing fan engagement that could be readily integrated into MLB's official application.

Inspirations

12 million people follow the MLB on Instagram, 12 million on X, 6 million on YouTube, and tens of millions of people have downloaded the MLB mobile app. How many of them are satisfied with what they see and feel engaged?

Like baseball fans, I like professional sports but information is often too dull or too much, hence the creation of Fan Frames. The goal is to save time while enhancing the fan experience by providing targeted excitement instead of continuous slop.

Relive your favorite sports memories thanks to Fan Frames' personalized text, audio, and image highlights. Generating personalized highlights should feel like opening a lootbox where the items you get please you every time.

I chose to portray highlights using MLB teams mascots instead of players to follow Imagen 3 responsible guidelines, but also because mascots are a key part of both the fan experience and team identity!

#GoogleMLBhackathon #BuildwithGemini #Dokine

Disclaimers

The application works as long as both the MLB API and Google Cloud models are available.
If the application is up but doesn’t work, make sure you've entered valid inputs and try again.
If the live application doesn’t work, it can be due to it reaching my budget quota. Follow the tutorial to host it yourself.
Text and audio highlights don’t change much for the same inputs but the image does because I chose to randomize several parameters like style, stances, and time of day.

Usage

Access the application by going to its assigned URL, select your desired language and team or player and day or season, click on ‘Generate Personalized Highlights’. After about 10 seconds, you will receive a text summary and an audio file. After about 10 more seconds, you will receive a beautiful illustrative image.

UI 1 UI 2

Building

I’ve built the application using only 3 fully managed elements (1 App Engine deployment and 2 Cloud Run Functions):

The App Engine deployment is a fully managed serverless platform that allowed me to focus on coding the Front End without thinking about availability, scaling, and more. Simply execute ‘gcloud app deploy’ and the web application is live worldwide!
The first Cloud Run Function is dedicated to fetching data related to the fan request from the MLB API and generating both the text and audio summary using several Google Cloud services: Vertex AI, Gemini 2.0, Translate, Text-To-Speech. Cloud Run Functions are fully managed lightweight computations solutions to respond to events, HTTP requests in this case, and execute code.
The second Cloud Run Function is dedicated to generating an image related to the fan request using Google’s Imagen 3 model.

The Architecture Diagram is available in attachments as an image:

The goal is not to get caught up in infrastructure management since Google Cloud takes care of it, the goal is to generate value out of existing data in the simplest way possible using Google Cloud cutting edge services.

Learnings

When it comes to technology, I got better at:

Writing and deploying a front end on Google App Engine
Writing and deploying a Cloud Run Function
Making Google Cloud services interact with each other
Prompt engineering text and image AI models (1)

When it comes to operations, I got better at:

Only using the required data to avoid waste of compute resources and model tokens (2)
Documenting the work produced for others to use it

When it comes to product management, I got better at:

Following the requirements rather than expending out of a given scope
Prioritizing items based on their value to meet a short deadline
Communicating about a product in a video format

1) I initially tried to generate complex images without taking into account the aesthetics too much, only to end up reducing the amount of information displayed and focusing on the appearance, especially emotions and style, of the ones that mattered most. Instead of writing ‘Win’ or ‘Loss’, make the subjects reflect the result using adequate stances. I kept 3 stances, 3 image styles, and 3 times of day to balance variety and quality.
2) I reduced the number of tokens tenfold without degrading Gemini’s output by only extracting necessary data when querying the MLB API using hydrations and fields. Even though Gemini models handle 1 million or more tokens doesn’t mean you shouldn’t count.

Challenges

Even though I had a rough idea of my goal at first, it took several decisions, trials, and iterations to end up with a satisfying application. The most important questions were “How much control do I give the fan?”, “Do I represent humans celebrating or the team mascot?”, and “How do I optimize for cost and efficiency?”. This is why I decided not to let the fan influence the content orientation (summary length, audio tone, image style), nor represent humans (mascots are universal), nor make the app store the files (audio and image) for now despite having tried it. Perfection is achieved when there is nothing left to take away.

Accomplishments

Building an application useful to many stakeholders in under a month feels rewarding: fans will have a better experience, Google Cloud will have a new example to showcase its technology, MLB will have a refreshing inspiration for its app, DevPost will have a quality project to showcase, and I will have gotten better at several things!
My favorite part was being able to generate quality images related to various input data while keeping the coherent style I wanted.

Limitations

Two of Google's latest products weren’t in General Availability during the Hackathon period, which could have significantly enhanced the fan experience: NotebookLM for a podcast summarizing a Season (https://notebooklm.google/) and Veo 2 for a video highlighting a key moment (https://deepmind.google/technologies/veo/veo-2/).
Despite being very complete, the MLB API wasn’t fully accessible: the centralized metrics for a given player (https://statsapi.mlb.com/api/v1/people/{player_id}/stats/metrics) weren’t directly available, which forced me to find a less convenient and less complete workaround.
Because Imagen 3 blocks generating images of celebrities and writing their name on jerseys (https://cloud.google.com/vertex-ai/generative-ai/docs/image/responsible-ai-imagen), I was forced to take a creative approach to represent the teams. One important parameter of Imagen 3 is person_generation, which value is dont_allow unless you get approved by Google through a form to use allow_all. This change drops the generation error rate from 50% to 0%.

Fan Frames will be what fans and MLB make it, it can stay as-is or be improved by anyone!
I may try to use NotebookLM to generate a season podcast and Veo 2 to generate a key play video once they are Generally Available.

Criteria

Technical, relevant and effective use of Google Cloud Services and MLB data:
- 1 App Engine deployment, 2 Cloud Run Functions, 4 Vertex AI API (Google Cloud)
- People, Schedule, Game, Hydrations, Fields (MLB)
Presentation, clear and engaging while showcasing the technical solution:
- Promotional 3 minute long video
- Project story DevPost post
- Public GitHub repository
Innovation, latest tech like Imagen and Vertex:
- Vertex AI (Chat, Translate, Speech, Vision)
- Gemini 2.0 (gemini-2.0-flash-exp)
- Imagen 3 (imagen-3.0-generate-001)
Creative, address a MLB challenge while enhancing fan experience:
- No equivalent available on the market
- Ready to be integrated in the official MLB application