Inspiration

When networking / talking with people, names and other info is typically forgotten very quickly. Having information readily available about the potential employers while you speak can greatly improve your networking experience.

What it does

Uses facial recognition to determine locations of faces in a live video feed and overlays info about detected people. This information is viewed in real-time on a VR display as an augmented reality pass-through. The system allows users to have the most relevant info in their sight while talking to others, reducing stress and increasing efficiency in communication.

How we built it

The frontend is a Unity 3D Android app using ARFoundation for base AR functionality. Facial detection is done by sending a JPG every 10 frames to the Azure Face API backend which returns bounding boxes of all faces. Next the bounding box centers are mapped into the 3D world to display info text around faces.

Challenges we ran into

The original tools we used (ARCore) did not allow for facial detection using a back camera (by design). We had to change the tools we used and develop our own solutions for certain tasks. Mapping screen space coordinates from camera to world coordinates was difficult to do reliably. We had access to limited hardware, mainly only having a VR headset instead of an AR one.

Accomplishments that we are proud of

  • Integrated many different tools to make application work
  • Able to recognize faces, do augmented reality and display visual information in a suitable way for a VR headset.

What we learned

  • Azure Face API
  • Coding Unity 3D games in C#
  • Google Cloud Platform
  • HTTP requests

What's next for NetworkSmartAR

Adding more displayed information about individuals. Adding voice recognition for names and conversation details being added to displayed information.

Built With

Share this project:

Updates