Inspiration
Low-functioning autism can make it difficult for families to engage with one another. In addition, it is hard for researchers to study the condition.
Jeff Munson has created a simple game using Kinect and the Unity engine to address these challenges. In order to promote engagement, he wanted to add a control panel interface on an Android tablet that a family member or friend can use to create the scene and adjust other settings.
In addition, we wanted to improve the game play visual experience.
What it does
The existing game, run on a laptop and projected/displayed on a large screen, displayed back the camera feed, like a mirror, with the addition of a number of colored balloons, which players could pop by moving their hands to intersect with them. It would load one of a handful of pre-created balloon locations.
During the hackathon, we created a control panel application that a family member or friend (the "partner") can use to create custom levels in real time. We also leveraged new built-in logic in the Kinect V2 to use a green-screen algorithm to replace the room with alternate environments.
How we built it
The basic game was written in Unity, using a Kinect user interface. We developed the control panel using Unity so that it can be more easily maintained without learning an additional language/environment and so that it can run on an Android or windows device. The control panel is run on a tablet and communicates with the main program using a TCP connection to convey the locations for balloons.
We leveraged new libraries with the Kinect V2 to show the player as a 3D model, rather than a 2D image like before. In addition, we used green-screening to replace the background with a forest model.
Challenges we ran into
Most of our team was new to Unity and the .net framework, so there was some learning required to implement these changes.
Accomplishments that we are proud of
Contributing to a product that will have positive impact for people with autism and the community around it.
What we learned
We learned more about autism, gaining more compassion for this population, and how games can be useful for connecting with them.
What's next for Interacting with Autism
Jeff Munson will continue developing the product. Some short term goals are to implement network discovery, since right now the IP address is hard coded. Long term visions include having a variety of games, automating upload of logs for research.
Log in or sign up for Devpost to join the conversation.