Inspiration: I was an F1 junior driver growing up, I love everything sports and I’m also obsessed with world models. When I saw what world labs (atlas) posted a couple weeks ago I had an idea for using it in sports and this is the closest thing I can get to that.
How it works: It ingests video data from YouTube, using transcript and crowd noise to auto source highlights for football, if anything else it allows for manual selection of highlights. It then allows for a freeze frame to be selected and runs it through H3 max to give the viewer a cool 360° view video playback before continuing the video onwards. Sound slowed down also to immerse the viewer.
How we built it: I was token maxxing on codex! Really though I created an HTML space to ingest video easily and used wispr flow to identify noise markers for highlights. Then set up an API call to H3 with a clever prompt to get the 360 perspective I wanted. I also firecrawled reactor (sorry) to get the design I wanted - UI is important!
Challenges we ran into: H3 is not thatttt good of a model in 4K, so it looks really cool but not to the level I’d like it to be.
Accomplishments that we're proud of: I’m proud to have something running in real time. It works, we can run it end to end in 1 minute. With more inference and better internet maybe I could get it down to 30 secs.
What we learned: that H3 max is not ready for production level viewing and inference just yet. Perhaps when world labs’ atlas is released that will be the answer here.
What's next for 360° world model clips for sports:
I had my YC interview just 2 weeks ago. They thought it was too ambitious for right now, waiting for inference to get to a place where this can run quickly enough that broadcast or socials viewers can really enjoy these clips right after they occur (near real time). I think World Labs Atlas will likely change that.
Log in or sign up for Devpost to join the conversation.