Flowchart Representation of all the components utilized.
We wanted to explore the applications of Virtual Reality for Social Good. People with spatial hearing disorder face a difficulty in identifying the direction of the audio source. This leads to them being unable to differentiate between voices from the crowd to that of people talking to them. We planned to come up with a fun VR experience to help them improve their spatial hearing abilities through cognitive training. Additionally, we save all of the data from each session for purposes of trend analysis and further research. This project can be taken many ways in the future. We envision it becoming an open-source research project that can not only generate a lot of new data on the spatial hearing disorder but help to treat the disorder.
What it does
SpatialVR provides a user-friendly VR platform for patients, doctors, and researchers to tackle the spatial hearing challenge. On one level, it acts as a medium for the patients to practice hearing exercises and improve their auditory perception through regular practice. Each user is allocated an anonymous unique ID which would keep track of the past records (if any) and thereby provide real-time feedback on their progress. The measurements are collected by the application and shared on a server which keeps track of the information of multiple patients. This data is then presented in a user-friendly manner on a console and it could be used by the doctors to provide appropriate medical feedback. Moreover, the collected data is presented in a detailed manner by classifying it on the basis of age and the average time it takes to find the audio source. It would also prove to be a great asset for the researchers to look for specific patterns in the consolidated data by making use of Data Analysis and study more about this challenge by considering it from different viewpoints (say age, gender etc).
How we built it
The VR experience was built using Unity3D and the Oculus Rift SDK.
The data generated by the users playing the game is sent over to a Node, Express, and Mongo backend for storing and processing. This data is then presented onto a web console with React and D3. Our aim is to detect patterns in this data and aid in learning more about it and thereby provide a better remedy for the spatial hearing challenge.
Challenges we ran into
1) None of us had experience with building VR, Unity or C# and had to learn all the stuff required to build this game in 36 Hours. 2 ) 3d audio in Unity 5.3 was a bit problematic but after going through tons of resources on the unity-forums we finally figured out the right settings to emulate 3d audio sources.
Accomplishments that we're proud of
1) A fully functional Unity game that helps people improve their spatial hearing abilities. 2) A cool web app built in MongoDB, React.JS, and Node.JS which talks to the Oculus Rift through RESTful endpoints and collects game data.
What we learned
We learned a lot of useful stuff especially the power of Unity3D and it's flexible libraries. We also gained knowledge by interacting with other hackers and supported each other's learning process by distributing tasks and exchanging information.
What's next for SpatialVR
We plan to reach out to researchers in the industry to collaborate and come up with useful cures for the people with spatial hearing problems. We would like to open source the VR experience and the data itself under the GPL license so anybody interested in helping us in this good cause could come forward and collaborate.