When you go to a party, a career fair, or a mixer on campus, you want to meet people that you are interested in talking to. However, it might be awkward to connect to other people for the first time. We thus aim to promote personal connections through augmented reality technology.

Making good use of HoloLens, machine learning, and cloud computing, the app allows you to navigate your social life by providing useful information about people around you. All you need is to simply cast your sight on their faces!

What it does

On the HoloLens, this AR app shows a profile page next to the person that is recognized in the sight in real time. The app detects human faces and sends photos from the camera to the Microsoft Face API. We trained a machine classification model on the Microsoft Azure cloud platform to identify the face of each individual. The app then fetches the bio and social media links to that person for the user. It also enables the user to text the other person through Twilio.

How we built it

It is a HoloLens app created by the 3D engine Unity and primarily programmed in C#. We trained the facial recognition model using the Microsoft Face API. We used MongoDB for the database that stores profile data collected from user uploads. The back-end is written in Node.js for efficiency. The back-end and database are deployed on Amazon Cloud for greater reliability. For privacy concerns, we incorporated Twilio as an extra layer between viewers and profile owners so that certain sensitive information (E.g. phone number) is hidden.

What challenged us

One of our challenges during the development was dealing with the latency. The AR experience is highly compromised with high system wait times. The Microsoft API requires a real-time communication between the camera stream and the analysis in the cloud. Therefore, we introduced asynchronized requests to speed up the process. The multithreading of nature makes the code tricky to write, and we are glad that we solved the problem at the end.

What we’re proud of

The app combines augmented reality, machine learning, and cloud computing in a novel way to enrich people’s social lives. To incorporate the many frameworks and aspects of the program was not easy, and yet we all enjoyed the hardcore learning experience that enhanced our comprehensive skills as software engineers.

What's next

We seek to improve the accuracy of the facial recognition models by organizing user data in a more reasonable and efficient way. In addition, we are considering a customized recommendation system by assigning individuals various tags based on their interests. In addition, we hope to add more interactive features to the app.

Share this project: