We were interested in combining virtual reality and machine learning into a single hack. When we saw that Firebase would be a sponsor, we tried to incorporate their services into our hack because we thought it helped bridge the gap between the two technologies really well.
What it does
For the VR portion, we put the user inside a modern art gallery. On the walls, we put 12-14 Bob Ross paintings that were taken randomly from or data set. Then when the user begins the training, a 25x23 grid of cubes change color based of the training. Each cube represents a neuron and the color change shows how much it reacted to the input (lighter = more reaction, darker = less reaction). After the training is complete, we take the data about the paintings and compress it down to 3 dimensions. We then plot this in 3d space inside the gallery. The user can then put their controller inside one of the cubes and a preview of which painting it represents appears. This allows the user to see how similar paintings are clumped together. At any point, the user can change the colors of the cube with the trackpad on the Vive controller.
We made a server that handled the machine learning and used firebase to know when to start/stop the training. It would also send data of the training to firebase, along with clustering information.
We created a Google Home app that allows you to start and stop the training with your voice, along with monitoring if it is running or not.
We made an Android app that allows the user to send messages into the virtual reality scene, enabling the user to communicate with the person inside vr. The android app can also start and stop the training with the press of a button. In addition, every time the training starts or stops, a push notification is sent to everyone who has the app installed on their phone, telling them of the update.
Just before we fell asleep Sunday morning (7am), we set up an extremely simple website and hosted it using Firebase. All it does is link to this devpost page. If we stayed up a bit longer we may have been able to flush it out a bit more. Link: deepreality.win
Challenges I ran into
Most of our challenges came from the VR and ML portions. Firebase was extremely easy to use and simplified a lot of the work.
Accomplishments that I'm proud of
We created everything we set out to do early, allowing us to polish our project and add features we came up with after getting feedback from others.
What I learned
We learned more about Unity, Machine Learning, and Firebase
What's next for Deep Reality
In the future this could be expanded to easily integrate with other types of data, not just paintings. .