Curveball Nexus BLR

Curveball- Nexus Component Architecture
Start of the UI
Realtime Information with highlights
Right pane - About Strategy pattern
Realtime highlights of Out during current video
Historical Web Search of between 2 players and details Agent output

Inspiration

We drew our inspiration from the Indian Premier League (IPL), a renowned cricket tournament celebrated for its thrilling and interactive fan experiences. The IPL excels in utilizing technology to connect fans with the game through real-time statistics, engaging commentary, and interactive features. By adapting these principles to Major League Baseball (MLB), we believe we can revolutionize how baseball fans enjoy the game. By taking up a challenge 3 "Real-time tool tip", we will significantly enhance the fan experience, making it more immersive and engaging.

Lessons from the IPL: Real-Time Engagement: The IPL keeps fans hooked with live score updates, instant replays, and interactive polls. We aim to bring these elements to baseball to make the viewing experience more dynamic.

Multi-Platform Accessibility: The IPL can be accessed via mobile apps, websites, and social media. We want our application to be available on multiple platforms to reach a wider audience.

Enhanced Commentary and Analysis: IPL broadcasts feature expert commentary that helps fans understand the game better. By using video analysis and AI, we can offer real-time insights and explanations for baseball.

Fan Interaction: The IPL encourages fan interaction through social media and live chats. We plan to include an AI-powered chat interface to create a more interactive experience for baseball fans.

Video Language Model: Integrating a video language model into our project will allow us to analyze live game footage in real time, offering insightful commentary and interactive tooltips. This technology will help us explain strategies, provide player statistics, and answer fan queries instantly.

Applying IPL Principles to MLB: --By drawing from the IPL and using advanced video language models, we aim to: --Enhance viewer engagement with real-time video analysis. --Make the game accessible with AI-powered commentary. --Encourage fan interaction through live chats on video of current game.

What it does

Our project consists of two main components: a frontend interface where users can watch live baseball video streams and interact through a chat interface, and a backend system that powers the intelligent features. The frontend provides fans with real-time information about the matches and important events. It includes an Agentic chat feature that analyzes the live video stream and allows fans to interact with the current happenings or inquire about historical events in the match. Fans can ask questions about strategies, get explanations of the current plays, and receive relevant information in a textual format. This creates a more engaging and informative experience for baseball fans, helping them stay connected and informed throughout the game.

How we built it

Our architecture focuses on both the user interface (UI) and the backend.

-We used the Agentic Framework to manage real-time interactions and implemented the Gemini-2.0 Experiment-Flash multimodel (Vision model) for video analysis.

-Vertex AI (Vector-Search) was employed for retrieval-augmented generation (RAG) to provide historical information.

For video extraction and segmentation, we used ffmpeg, which allowed us to split the video into 30-second intervals for detailed analysis.

-The frontend was built using React, ensuring a responsive and interactive user experience. The backend processes the video content and manages the AI-driven interactions, creating a seamless integration between the live video stream and the chat interface.

Architecture diagram

Curveball Nexus Component Architecture!

Explanation of Architecture

Curveball Nexus Component Architecture," is a system designed for managing tasks and interactions within a software environment, possibly related to user interface (UI) interactions and agent-based tasks. The architecture is divided into several components which interact with each other:

UI Interaction Component: This part of the system handles the front-end user interaction and is composed of several elements:

Frontend Tools: Includes tools like React Icons and Axios.
Frontend Components: Contains components such as App.js, Chat.js, and Header.js.
UI Components: Elements like Chat Agent, User Stream Video, and UI Display are mentioned here.
Agent Task and Tools: This area is to manage the backend or server-side operations and is broken down into: Agents: With elements like comp.py, _agent, and rag_agent.

Backend Tools: Mentioned is a tool named analyze_video.

Tasks: Includes various tasks like task_rv_video, task_va_ai_filter, task_rv_video, task_search_vertex_db, task_mlb_info_gather, and task_gen_ans.

User Entered Agent Queries: This seems to be the input mechanism where users can enter queries that the system will process.

5.Real-time / Historical: There's a central node that suggests the system can handle both real-time and historical data.

6.Data Flow: The architecture shows data flow between the components, with arrows indicating the direction of data transfer.

7.External Integration:

-On the right side, there's a note that says, "Go to web video tools to extract Realtime and vision API," which suggests integration with external video analysis tools.

-Below that, there's a reference to "Google Vertex AI (Vector Search)," indicating the system use Google's machine learning platform while video will be segmented 30sec each frame on live and then use Vison model to extract the information which will be loaded to Vector Search.

-There's also a mention of "UI Agent Chat," which could be the interface for real-time communication or processing of user queries.

-Start: An icon representing a rocket labeled "Start" is shown, which could symbolize the initiation of the process or the entry point of the system.

This architecture is a conceptual representation of how different components within a software system interact to provide a comprehensive service, possibly for tasks like video analysis, real-time data processing, and interactive user interfaces. The logic behind the system is to integrate UI components with backend tasks, leveraging machine learning and video analysis tools to process user queries and deliver real-time or historical data.

Challenges we ran into

We encountered several challenges during development.

-Managing live streaming required handling network-related issues and downloading and segmenting the video efficiently.

-Replicating live video was challenging, so we manually synchronized the live data feed with the video segments.

-Ensuring fast interaction speed and video processing was critical, but Google Cloud credits limited our ability to perform extensive performance testing.

-Transitioning from Azure to Google Cloud involved a steep learning curve, which took additional time to overcome.

-Hosting the application also posed challenges due to credit limitations and unreliable connection between frontend and backend python services on Google Cloud, which delayed some aspects of our deployment.

Despite these hurdles, we successfully integrated the necessary technologies to achieve our goals.

Decisions and Workarounds:

-Due to the 3-hour long video, we have decided to use only the first 30 minutes for this demo or analysis.

-We will showcase one question as a demo that provides real-time information about the video. The remaining questions will be explained in the screenshot due to the time limit on the demo video.

-Instead of streaming the video, we used a downloaded version and displayed it like a live stream to avoid network-related issues.

-The video was split into 30-second segments and delayed by 90 seconds to show the video. There are some blackouts during transitions between segments due to limitations of React, which we plan to improve in the future.

Accomplishments that we're proud of:

-Real-Time Highlights: We achieved the ability to generate real-time highlights during the match, complete with video explanations of each clip.

-Agentic Framework Integration: Fans can interact in real-time, and if they miss the first few minutes of the match, they can receive a summary to catch up quickly.

-Historical Analysis: Our system can extract and analyze historical events in real-time, providing a comprehensive view of the match.

-Strategy Interaction: Fans can inquire about current strategies and counterstrategies being employed in the game.

-Live API Integration: We successfully integrated major events from the live API, ensuring up-to-date information.

What we learned

--Video to Text and Vision Model Usage: We gained useful experience in converting video content to text and using vision models for real-time analysis.

--RAG Integration with Vertex AI: We introduced a new experience in Google Cloud by integrating RAG with the Vector Store of Vertex AI as another RAG dB, although it comes with its own challenges.

-- Gemini-2.0 Model: We found the experimental Gemini-2.0 multimodal model to be very effective and better than other vision models like LLAMA currently available in the market.

-- Google Cloud Integration: We learned to connect frontend and backend services on Google Cloud, which improved our cloud deployment skills.

What's next for Curveball Nexus BLR:

-Internal Implementation: We plan to apply a similar approach for vision-based use cases within our company, leveraging the techniques and technologies we've developed.

-Plug and Play Solution: We aim to refine our Agentic framework to create a plug-and-play solution for baseball fan engagement on MLB platforms. This will make it easier to deploy our interactive and intelligent features across different baseball fan applications, enhancing the overall fan experience.

Testing our "Curveball Nexus" APP

you should start the video by pressing play button and then in Nexus AI Assistant you can ask the below questions.

Background About Video Played:

The New York Yankees and Los Angeles Dodgers have a long history of rivalry in baseball, starting from their first World Series in 1941. Their games often have excited crowds and sometimes even on-field fights, showing how competitive they are.In this special game called the World Series, the Dodgers and the Yankees played to see who the best team is. They played five games, and whoever won the most games would be the champion. In the fifth game, the Dodgers did a great job! They hit the ball well and ran around the bases to score lots of points. The Yankees tried their best too, but the Dodgers scored more points. So, the Dodgers won the fifth game and became the champions of the World Series!

*User to Ask Question: *

*Type 1: Real-Time Query: *

  -User query: - Provide me insights as to what is happening in the game right now also mention any strategies used.

*Type 2: Historical Information Query: *

 -User Query: - Provide me all the major events that have occurred in the game so far.

 -User Query: - Provide me all the outs that have occurred until now along with the names of the players.

*Type 3: Google Search RAG: *

 -User Query: -Provide me 2024 stats for Shohei Ohtani vs Gerrit Cole whenever they have faced each other. Which player has performed better over the other and what can either player do to win against the other?

 -User Query: - How many homeruns occurred in the past segments?

GitHub link:

https://github.com/himanshunanda22/CurveBall-Nexus-BLR-MLB-Insights

Disclaimers:

-The video is from YouTube, which is free, and credit goes to YouTube.

-Our company or any entity belonging to it is not responsible for any copyright issues, as we have used only public information.

Google Cloud Credits issue:

Our app has received significant usage, and we have exhausted all our Google Cloud credits. To enable the jury to test our live app, additional credits should be added to our Google Cloud account.

Built With

crewai
ffmpeg
flask
gemini-2.0-experimental-flash
geminivision
google-cloud
mlbstats.api
react.js
scrapewebsite
serper
vector-search
vertexai

Submitted to

Google Cloud x MLB(TM) Hackathon – Building with Gemini Models

Created by

I worked on using React to build a smooth and dynamic UI, ensuring seamless user interactions. I was also responsible for developing and optimising the frontend and its integration with the backend for better synchronisation and responsiveness. Since this was my first time dealing with complex state management and API integrations on this scale, I faced several challenges. However, overcoming them helped me improve my understanding of React, enhance my problem-solving skills, and gain valuable experience in building efficient frontend applications.

Himanshu Nanda
During the hackathon, I led a team to develop a video segmentation solution, managed tasks for timely completion, and integrated Google Cloud's Gemini 2.0 for advanced vision capabilities. I guided the UI design, articulated necessary changes, and tested the application, resulting in a successful and innovative project.

Dr. Prashant Ramappa
I worked on solving the problem of utilizing Gemini to generate insights and develop crucial strategies, particularly in the context of making use of live MLB stats. Additionally, I took charge of deploying both the backend and frontend, ensuring seamless integration and functionality. Since this was my first time deploying on Google Cloud, I faced several challenges. However, each hurdle became a learning opportunity, allowing me to deepen my understanding of cloud deployment, refine my problem-solving skills, and gain valuable experience in applying AI-driven insights to real-world scenarios.

Gagan Yadav
I designed and implemented the Agentic Chat system , an AI-powered feature designed to enhance fan engagement by delivering real-time insights from live video streams. Fans can interact with the game by asking about current plays, strategies, and past events through an intuitive chat interface.
To achieve this, I integrated Gemini 2 Flash exp for real-time video analysis, FFmpeg for efficient video segmentation, and Vertex AI Vector Search for context-aware retrieval of past events from the ongoing game—allowing fans to revisit key moments instantly. Additionally, I incorporated website-based RAG to handle queries related to player stats, strengths, weaknesses, and player comparisons, ensuring fans have access to in-depth analytical insights.
Working with Vertex AI Vector Search was a steep learning curve, but an exciting challenge—especially as this was my first hackathon. Overcoming technical hurdles in sports analytics and refining real-time interactions made the experience incredibly rewarding. Overall, it was a fun and enriching learning experience.

DRUVA Hegde