We thought the HoloLens was a very cool piece of technology, so we really wanted to work on a project using it. From there, we went through many different ideas, ranging from real-life Tinder at parties to augmented reality navigation systems until we realized we wanted to create a project with an aspect of social good that was also completable during the hackathon.
What it does
SocialQ is an Microsoft HoloLens augmented reality interface built to assist those affected by Autism Spectrum Disorder (ASD) with social cue recognition. When users find it difficult or overwhelming to interpret the sentiments of others, they can activate SocialQ with a hand gesture, which interacts with an API to provide them a brief interpretation of a selected person's facial sentiment. The interpretation is direct, but not intrusive. SocialQ functions not just as social "safety net", but as a constant (but subtle) reinforcement tool for users who need practice recognizing different types of facial expression.
How we built it
Our application relies primarily on the Windows Holographic SDK for Unity and the Microsoft Cognitive Services Emotion API. Our interface was developed in Unity, using integrated dynamic text meshes and image body-locked holograms. For input, we implemented existing HoloLens air tap gestures, which trigger photo capture of the user’s current view. The backend is a C# script integrated with Unity’s WWW framework to send the bytes from the captured photo to Microsoft Cognitive Services. On top of the air tap gesture, we implemented voice commands using the Unity Windows Speech Recognition library, which the user can use to ask questions.
Challenges we ran into
As usual, it seemed that a disproportionate amount of time was spent on figuring out how to use the hardware and APIs. Because our access to the HoloLens was limited, we spent a lot of time perfecting the user experience "hypothetically" using an emulator, and even then having to go back and tweak it after testing it with the real thing. It was a particular challenge to save picture data onto the HoloLens itself and send it to the API properly - this took us many hours. This proved to be our main challenge. Integrating asynchronous coroutines in Unity to make web API calls is very difficult, and there was little literature on the subject. We thus developed our own class to handle such calls, which ran on a separate thread as to not interfere with the front end animations and code. There was also some weirdness with data structures in C# and, particularly, the custom data structures, such as JSON Objects, with Unity which cost us some of our momentum. Thankfully most of these challenges were to be expected, and only serve to prepare our team for 3D Augmented Reality development in the future.
Accomplishments that we're proud of
It seemed that most of our celebration came from smaller things - fixing an issue with scope, figuring out why one of the packages wasn't installing properly, or getting the photos to save on the device. However, what we'll ultimately celebrate is that each one of those small victories was able to accumulate into one cohesive project that ended up working predictably and properly. We're particularly proud of the social impact of our product, too. We are excited to present a possible solution to benefit the approx. 1.2 million children who suffer from autism in our country.
What we learned
We learned what hacking really is. In using unfamiliar products and APIs, we encountered more silly debuggy / hardware / installation errors than perhaps ever before. But, there is a value in that type of debugging that feels more like cleaning a room than beating an opponent in chess. The real world is made up of moving parts, not abstract algorithms, and practical frustration is something that should never escape computer scientists. Cal Hacks taught us that something as humbling as a 6 hour struggle with a toggle switch GUI can make us better hackers.
What's next for SocialQ
We're not sure. Looking into Stanford's Autism Glass Project showed us how many different approaches there are to accessibility research. Providing a straightforward platform for ASD patients was gratifying, but it might be a logical next step to investigate potential expanded capability of the product. Expanded capability could include recognition of other nonverbal cues, such as gaze or body language.