We were inspired by a team mate who had travelled to Africa to teach students about technology and was frutstrated by the limited number of resources in the students native language. We were inspired to create a tool that would help educators world wide communicate with students regardless of their location, culture or language.
What it does
The project uses inexpensive Raspberry Pi 2 computers, Windows 10 IoT Core, and live translation services running in Azure to transmit translated text to a remote device that can then project a 3D image for the students to view. The 3D view is rendered dynamically using Unity, and the translated text is then converted to spoken word using the Cortana API.
How we built it
We built the 3D projection using Unity and published the display to a Windows 10 IoT Core UWP app. We then display that on a monitor that has a "Pepper's Ghost" inspired pyramid placed on top of it to create the feeling of a 3D image floating in space.
The text translation is done using the Text-to-Speech APIs built into Windows 10, then sends that data to a Project Oxford translation service running in Azure. The translated text is returned, and sent over the nextwork to the remote Raspberry Pi where the 3D display is rendered and the translated text is spoken using the Cortana API.
We built it using Visual Studio Community Edition 2015, Unity Community Edition, and a lot of cutting and gluing of Acrylic.
Challenges we ran into
Geometry. Turns out cutting a 4 sided pyramid out of Acrylic is a lot harder than it seems. In addition we ran into issues with the Raspberry Pi being temperamental about the USB port the headset was connected to.
Other challenges had to do with us all pushing ourselves to understand the tools and platforms we chose.
Accomplishments that we're proud of
Geometry. We finally got the pyramid made! All in all it is a pretty impressive collection of technologies to pull together in a single hack. Raspberry Pi, Windows 10 IoT Core, Universal Windows Platform, Azure Machine Learning, Speech-to-Text, Text-to-Speech, Pepper's Ghost, Acrylic Fabrication, Unity Multi-Camera Views, and Slam Chopping code! We are proud to have envisioned the project in the first place, and hope that you are as inspired by it as we were inspired by our teammate teaching in Africa!
What we learned
Geometry. No seriously. That was a lot harder than it should be for professional developers! We learned a lot about the various APIs including Raspberry Pi, Windows 10 IoT Core, Universal Windows Platform, Azure Machine Learning, Speech-to-Text, Text-to-Speech, Pepper's Ghost, Acrylic Fabrication, Unity Multi-Camera Views and Particle Systems.
What's next for HelloHolo
We would love to be able to render a full size person. In addition, we'd like to extend it to include the ability to remotely sense and control the environments on both ends using a variety of sensors and actuators.
Most of all, we would love to be able to give it to somebody else as they travel the world to educate others.