SnapShot

Inspiration

With the recent NBA playoffs, we as a group found ourselves watching a game together where we realized the vast difference in our knowledge of basketball as a sport. We thought that this was probably a common scenario amongst groups like friends or couples. Combining capabilities of AR with generative LLM's we figured we could create an extremely helpful tool that could answer questions about a game as it happens, instantly answering any questions a new watcher may have about whatever is happening on screen as it unfolds. But for all levels of basketball, it can remain highly useful to pull desired statistics or historical information and generate custom displays of relevant data.

What it does

SnapShot uses AI video recognition from live camera footage via the Snapchat Spectacles to recognize live NBA games. From here, it generates movable displays of crucial information like the box-score and play-by-play history, and crucially allows voice interactivity to ask questions at any point in the game that uses the camera feed for context of the game.

How we built it

Our front end uses Lens Studio with TypeScript for object scripting and user interactions on Snapchat Spectacles. The backend combines Python's NBA API library with MongoDB for data storage. A Flask webserver exposes endpoints for data access, with NGrok tunneling connecting our local environment to the front end. The webserver automatically refreshes so that data's remain up to date live. Matplotlib generates visualizations from retrieved statistics, which are requested and displayed through the Lens Studio interface when users interact with the application. Google Gemini's API is used to process the user questions, taking in the camera context and return answers.

Challenges we ran into

By far our biggest challenge was our lack of experience with the Snapchat Spectacles and their development software, coupled with the lack of clear documentation. With none of us having ever used Lens Studio, our understanding of how to code things like objects and interactions was limited to the little code available in the documentation and live demo. Even LLM's were largely unhelpful given their lack of familiarity so we had to do a lot of cross-referencing with existing projects and w as overall and extremely time-consuming process that brought about many roadblocks that led to being completely stuck at times.

Accomplishments that we're proud of

I think we were very proud of being able to navigate through these challenges and still end up with a working interactable product that has live capabilities as was our initial goals. Despite the major obstacles we faced, we were able to solve many issues like one of not being to pull an image from an http request with practically no resources but our ingenuity.

What we learned

From this project, we've certainly learned a lot about being adaptable and finding solutions to difficult problems, even if it is simply through unrelenting trial and error that eventually bears fruit. Furthermore, we strengthened our value of persistence where at times we all thought our task to be impossible or futile, we powered through and our unwillingness to give up ultimately brought us to accomplish the goals we came in with.

What's next for SnapShot

What we have built is perhaps merely the beginning of SnapShot which has so much more potential. The next features we'd want to implement include things like user customizable graphics to display, perhaps with the help of LLM generation, and then full dashboard presets of the graphics users want. Beyond that, there are many more complex things that could be done like hand-tracking to identify specifically where on a screen users are pointing to and even creating some type of highlighting of things on the screen to point out relevant things to users.