Stats on Sight

Live in action!

Inspiration

For our second time around at nwHacks, we wanted to be adventurous and try our hands at a new piece of tech -- augmented reality. We're all big sports guys and have this problem where we're watching a game at a bar without sound, sometimes far from the screen so we're not getting the usual stats from the announcers over the course of the game. With Stats on Sight we wanted to make live game statistics more digestible from our phones.

What it does

Stats on Sight uses your phone camera to capture a live NHL broadcast, determine which teams are playing from the screen and then it displays a variety of statistics in augmented reality around the screen.

How we built it

We built an iOS app with Swift that identifies a vertical plane (the screen you're pointing at) with ARKit. Then, also with ARKit, we crop a rectangular image of the NHL game from the camera view and upload it to Google Cloud Storage via our Node.js API hosted with Google App Engine.

We then pass the image through Google Cloud's Vision AI and identify which two teams are playing. Given the two team names we can pull live stats from the NHL's public API and return that to the client, displaying the stats in an AR dashboard.

Challenges we ran into

ARKit was entirely new to our team. The documentation was really poor and hard to navigate which made it challenging to work with. We were also not expecting to be fooling around with 3D geometry or doing matrix transformations at 3 AM.

At some point in the middle of the night, Google Vision AI was having a lot of trouble processing our images for some reason which stalled our progress.

The most technically challenging part of this project was anchoring the stats that we wanted to render with ARKit to the actual laptop screen. The reason this was hard was because the "anchor" that we attach our AR stats to is the live-broadcasted game which is constantly changing.