F1-Vision

Inspiration

As sporting events like Formula 1 become more expensive and exclusive to the physical experience, many fans are unable to immerse themselves in their favorite sports. Watching F1 on TV every weekend can become boring and repetitive. People often feel like they're missing out on the real excitement and thrill of the sport.

What's more, F1 is not just about cars racing around a track, it's about strategy, quick decisions and minute technical details that make all the difference. At the same time, there is a growing shift in the sports world towards data analytics and immersive technologies. As more teams, analysts, and fans alike rely on detailed statistics to understand games, the demand for a more engaging and analytical experience has grown. The best way to change that is to make it more interactive. Make the game feel around you, immerse you in those moments. That is why we created F1-Vision, the best way to experience F1, not watch it.

What it does

With F1-Vision, we bridge the gap between live races and data, putting personalized control at the fingertips of the viewer. Watch live, location-based races and experience interactive maps and dashboards, along with any data you can imagine, from tire wear to sector times to radio messages, live from the race. We also provide exciting and insightful commentary and in-app race alerts so you don't miss a thing.

How we built it

We used Python to scrape different types of F1 data. We parsed it and added features to the dataset like relative time (time that starts when the race starts) and then stored it in files in the json database. We then pulled those into XCode where we developed on VisionOS for the Apple Vision Pro using Swift to develop several UI elements such as the dashboard and notifications, radio and video commentary. The notifications were stored in a file and indexed by timestamp, which was called and displayed by the main running loop.

The radio is the interaction of the driver with the mechanic and the audio is shared publicly, we put the audio and made UI elements around it. The video commentary is done in a pipeline of 2 generative ai models and 1 lipsync.so model. First, we gave the llama model context (some of the things that happened around the event during the Formula 1 race), and it then provided commentary based on the events it was shown.

We then used a natural language text-to-speech model called Whisper-Speech, which basically works as a voice replicator to copy the voices of famous commentators. We used this model to convert our text commentary into audio.

To make it more interesting, we decided to turn it into video using sync.so to put it on another character of our choice. Our AR/VR simulation is a lot of 3d model design in blender and a lot of mapping for the Bahrain F1 track. We used interpolation and the light design system to make the simulation more realistic and visible. The simulation then follows points in 3D space that it fetches from the database and simulates.

Challenges we ran into

Scraping different types of data with different relative times was a challenging task. Graphics with Vision Pro and interactive 3D models also required new skills and understanding due to a lack of documentation and support. We also faced many challenges in integrating our data systems into Swift, especially because of the high speed of the data flow. Through countless hours of tweaking, we figured it out and decided to create a running loop that acts as our internal system clock, allowing us to smoothly display data on the dashboard.

Accomplishments that we're proud of

Learning Vision OS and working with a cutting-edge technology that had not been used thoroughly before. Smooth implementation of VR/AR technologies and 3D models, we also added animations and had to perform complete data Complete data scraping and analysis. Working with events and creating a system to implement high speed data flow. Creating a high quality UI experience with animations.

What we learned

Implementation of Vision OS and the technical difficulties of using it. Swift, Reality Kit SDK Json structure and concurrent use of json data. 3D modeling and manipulation of various files. How video generation models work, speech replication using Whisperspeech.

What's next for F1-Vision

With F1-Vision, we're changing the way people watch F1. We're creating a new way for fans to enjoy the game. We plan to add more maps, replays, camera angles and better views. We also plan to improve our scoring system. We've talked about using machine learning models to predict different things. We've also talked about integrating other community-based systems to interact with the community.