HoloHacks

Our website
Our Custom Vision Model Performance Statistics

Inspiration

On average, 65% of people think that artificial reality will become a part of everyday life. link

After attending the HoloLens Demo hosted by Microsoft, our team observed that the current virtual keyboard contained a cybersecurity risk that could be exploited! Assuming the role of white-hat hackers, we wanted to make Microsoft aware of this potential threat and propose solutions that could be pursued in the future.

Even though the field of virtual/mixed reality continues to grow, we are not yet sure how many devices can--and will--be hacked. By exposing this vulnerability now, we have the potential to save the millions of people who will benefit from using virtual/mixed reality!

What it does

When typing into Hololens' Virtual Keyboard, our hack intercepts the user's input. Using this, we can parse together passwords, messages, and other sensitive information with nothing more than visual contact to the target.

How we built it

Client-Side: Javascript, HTML5, CSS, React, Microsoft Hololens, Microsoft Cognitive Services Vision Solutions Template Server-Side: Microsoft Azure Custom Vision

Challenges we ran into

Distinguishing user input - especially when a password had letters that were close together on the virtual keyboard
Because we did not have access to a Kinect, we were not able to use Microsoft Project Gesture to accurately identify gestures, especially when taking videos or images
Processing big data -it was time-consuming to take in, parse, and tag over fifteen images from every letter

Accomplishments that we're proud of

We are proud that we were able to detect a genuine cybersecurity risk, create a demo that represented the risk, and propose potential solutions to mitigate the problem. We were also proud that we implemented hardware and balanced diverse skillsets for our project.

What we learned

We learned how to use Azure Custom Vision's Object Detection Services to detect unique hand signals. Previously, we had only used Azure Custom Vision to conduct a basic A/B test, and the challenge of determining different hand gestures that were fairly similar to each other was a time-consuming process.

We learned how to take a picture from a website and return its result. This involved using Postman to post our images to our Azure Custom Vision Prediction API. Furthermore, we used Microsoft Cognitive Services Vision Solutions Template to post a video of us typing the password in real-time.

Finally, we learned about the business challenge of not only presenting a problem and its demo but also proposing potential solutions. We wanted our project to exist under a realistic business scenario, and that involved having an impact even after TreeHacks had ended.

What's next for HoloHacks

Use linguistic analysis to determine where the virtual keyboard is, increasing the accuracy of guessing the password
Use Microsoft Project Gesture to better target the unique gestures used to press each key
Increase the dataset of potential users to make the project more inclusive and accurate. (Our dataset currently includes the hand gestures from one right-handed person. We would need thousands of inputs to make our project scalable.)

Stanford McCoy Family Center for Ethics in Society - Most Ethically Engaged Hack

As our team was watching the demonstration of Microsoft’s Hololens 2, we noticed a potential security vulnerability in the digital keyboard interface used by the Hololens. This is a significant ethical problem as this information could be used to exploit the millions of virtual and mixed reality users in our near future. In addition to learning passwords, this vulnerability could be used to parse any sensitive information that a stakeholder typed using their Hololens. Furthermore, stakeholders who only have one password for all of their accounts could be at risk of having all of their accounts compromised. Our team recognizes that Microsoft and its customers are key stakeholders in ensuring that their data is protected when using the Hololens 2.

Traditionally, hacking an object such as a Hololens would be done for unethical reasons. In our hack, we took on the ethically-focused role of white-hat hackers to demo what hacking the Hololens Virtual Keyboard may look like. Instead of hacking for our personal gain, we hacked in order to make Microsoft aware of the potential security risk and to provide solutions to the problem. The steps we took to create our demo involved training images to detect when a stakeholder was virtually clicking a key. In a normal circumstance, we would have made this data public, but knowing that this could be dangerous information in the wrong hands, we have chosen to keep these images private and not push them to the Github repository or display them publicly. Finally, we hope that our project brings awareness to the potential security threats associated with the Hololens. Our stakeholders would never let someone peer over their shoulder when typing sensitive information into the computer. Similarly, they should not let people observe their hands motions when typing sensitive information into a Hololens.