Inspiration

Scavenger hunts are a fun group activity. You race against your friends to find a set number of items faster than them.

The only problem, is that you have to be together in order to play it. What if we could change that?

What it does

ScaVision Hunt uses Google Cloud’s Vision API to detect items in your surroundings and Firebase for real-time communication between devices.

Every device receives a prompt that tells them what they have to look for. They are then asked to search for it in the real world and snap a picture with their phone. The image is uploaded to Firebase Storage, and then passed to Vision API to determine it’s contents. This allows anyone, anywhere, can play together, as long as they have an internet connection.

How I built it

The project was built almost entirely with Firebase and Vision API - from hosting, to real-time databases, to cloud functions.

This project is a web application that uses JavaScript, node.js, and HTML. The project works by displaying a live camera feed on the user’s screen. When the user takes a picture by tapping the link, the current frame is grabbed and sent to Firebase Storage. Using Cloud functions, a tigger was set for whenever a file was uploaded or changed in Storage. This cloud function forwards that image to the Vision API, which then uses the labels feature to detect items in the picture. Any item with a 0.5 or higher score is returned.

This was unfortunately as far as I could get on my own, as I am currently having issues writing the returned labels to the real time database. After this, I had planned to run another cloud function that grabs the current item to search for, and run the returned array of labels against it to look for matches. If a match is found, the script will indicate that in the real time databases, and the devices will show a message showing who found it, their picture, and how many points they earned.

Challenges I ran into

As you can tell, I didn’t finish my application. I ran into some issues with the real-time database in Cloud functions that prevented me from writing to it. Everything that I have been able to accomplish so far has been published to both Firebase Hosting and Github. However, with extra time, I plan to fully finish it.

Accomplishments that I'm proud of

I’m mostly proud of the fact that I was able to learn how to use machine learning and a database API through Firebase and Google Cloud. I had never even dreamed of being able to perform machine learning without Google’s set of tools.

What's next for Scavenger Hunt

I hope to fully finish this application once I can work out my issues with Firebase. There was an immense amount of interest in my project, especially from students who had to sit in the long bus rides coming to SB hacks. I can imagine this being a great impromptu team building game if I can release a stable version.