Our team designed an application for the Microsoft HoloLens that could help individuals engage with social media in a healthier way. We envisioned an AR Buddy (based on ShellHacks "Shelly") that would help users engage with their neighborhoods while linking posts and feeds from social media platforms to physical spaces. Our functionality was to include:
- An AR buddy that the user could talk to
- Spatial Anchoring of social media posts to GPS locations using Echo AR
- AR billboards and donation requests
- Real-time safety measures, such as activating video and calling 9-1-1 on user command
Essentially, this application was an effort to combine virtual and physical social spaces into a coherent whole, while providing a buddy to help the user navigate the mixed reality world.
What it does
Currently, our application can display a virtual buddy hologram on the Microsoft HoloLens. It can tether the buddy to the user, and provide the illusion that this buddy is navigating the physical world alongside you. It can take user voice input. It can anchor a hologram to a physical location, and provide persistent AR features mapped to real-world spatial features. It also is capable of occlusion, providing much more interesting and believable AR effects. The AR buddy also sends text reminders to remind the user to take care of himself/herself : Did you sleep 7 hours ? Did you go for a walk ? Did you take a break from coding ?
Take your Shelly friend with you anywhere, even on your phone using EchoAR ! Check EchoAR buddy on here --- > https://github.com/inessaid/echoarbuddy
How we built it
We built the HoloLens application in Unity, and did the scripting in Visual Studio. We built the package for Windows Universal Platform and exported it to the HoloLens.
We built Shelly, the AR buddy, utilizing the standard Maya - Substance Painter - Unity workflow. Our designs were based off concept art for shelly, and the eyes are based off "Cappy" from Super Mario Odyssey.
Challenges we ran into
The Microsoft HoloLens 1 is a prototype device. It is prohibitively expensive and has a small user base. We experienced a number of problems working with the device stemming from lack of support. This was the first time we ever experimented with the Microsoft HoloLens and we were unaware of the compatibility issues beforehand. It is also difficult to set up this device, requiring a lengthy and confusing pairing process.
We also had trouble working with Microsoft Azure. We intended to provide a much more robust Spatial Anchoring system, but the tutorials and support for the product were limited and difficult to follow. We believe that the HoloLens 2 is more widely supported, and we will likely use the updated device in later projects.
Accomplishments that we're proud of
Working with the Microsoft HoloLens, although frustrating, was extremely interesting. It is one of the most futuristic and boundary-breaking devices any of us have ever used. The experience of being in a truly mixed reality where digital and physical worlds co-exist simultaneously was shocking and thought-provoking.
Although we did not achieve the full functionality we designed for, we did build a hologram application with a custom AR buddy. We learned about the HoloLens production process and gained many ideas for future innovations. This Hackathon was a success.
What we learned
Primarily, we learned how to setup the HoloLens device and integrate custom models into the HoloLens environment. We learned how to implement Azure resources into a HoloLens project. We learned how to add voice recognition services in the AR environment.
Perhaps more importantly, we learned about the capabilities of the HoloLens device, and gained many ideas and useful skills to use in future projects.
What's next for ShellCare
Ideally, we would like to finish the project and build out the rest of the functionality.