Social Cues for Autism - Game/Data Analysis

VR game startup
First question.
In game.
Incorrect, flashes "wrong" and does not spawn cats.
Focused object report (if the user looks at the slightly smiling face for a long time, maybe bc they're confused)
Unity Interface

Inspiration

Helen: When my friends come from work, they talk to me about their day. One is a SD substitute teacher and another is a K-2 special education teacher. Sometimes they tell me about their children - who was acting up that day, what behaviors they had trouble with, which incentives worked (if any at all).

For the substitute friend, he often works in elementary school, where autistic children's behaviors often go unchecked because they're either 1) not dealt with at all or 2) automatically graduated to the next grade, to new material and environments that they're still not ready for. We would discuss this every night, wondering what the children needed and what a substitute, in his limited time, can even do.

For my K-2 special ed teacher friend, he's been with them for a few years. He's able to manage their progress over time and, unlike many teachers, tries to teach them proper behaviors. This means finding their problem behaviors, incentives, and so on.

In both, managing and teaching behaviors come first. There's only some time leftover to teach things like math or reading.

So when the last woman in the ceremony came to the stage and talked about her work with autistic children, I immediately knew what I wanted to use the eye tracker. It would be a godsend tool for teachers to find out exactly which social cues their students had trouble recognizing and why. I didn't know if I'd participate - I've gone to a couple hackathons before but never did anything. I even did this for SDHacks last year. But I'd always heard Unity was easy, and I'm finally taking my first CS class this quarter, so I went for it.

Anshul/Karishma: During the opening ceremony, one of the speakers mentioned about the reach of technology and that one should be able to experience technology in 3D and not just 2D, like we daily do. There was another speaker talking about how one could use technology to help kids with Autism. This gave us the inspiration to build this healthcare application to cater to kids with special needs. It focuses on teaching them basic social cues, which is difficult to do in real life, because of social constraints.

What it does

Question Game I created a very basic Unity game where the user answers questions on the screen about recognizing facial cues. If the user answers correctly, the game spawns a cat friend. If the user answers incorrectly, the program lets him/her know. Anything that the user looks at for a long time is presumably a source of difficulty - e.g. looking at a confused face for a while, can't figure it out. So the program takes this into account and reports it to the teacher in an end report.

Parts that of Question Game that we wanted but not finished: I wanted to make it so that we can detect specific parts of the face - e.g. user never looks at eyes, only at mouth. This would help the teacher train the student to look at these sections in their behavior lesson plans. The program itself would use the student's difficulties and adapt its questions to it - e.g. more pictures with bigger/more expressive eyes and small/covered mouths. The program was eventually supposed to have different modes/difficulty specific to the student. I wanted to use GenomeLink for this, so I can generate levels based on the student's reading ability and personality. An easily neurotic student, for example, wouldn't react well to many angry faces.

The VR App: This VR application teaches kids with Autism about Proximity in any social situation. We basically try to answer, how close is close enough? How far should one stand while talking to someone?

How I built it

Helen: Both the ideas were prototyped in Unity in C# and for eye ball tracking the Tobii eye tracker API was used. I made/repositioned objects and created C# scripts for them. Helen also attempted Javascript for GenomeLink for a bit, but could not get it to integrate it with Unity.

Challenges we ran into

Helen: 1) I've never used Unity or C#. I'm taking my first Java class this quarter, so I knew a little. I went to the workshop and got lots of mentorship help for these. There were object and components that even left the mentors baffled. 2) Tobii eye tracker was difficult to mount and use. This is only because I have a computer that doesn't have a good mount for Tobii. I constantly had to readjust the tracker and myself to use the program. I luckily found tape, which partly fixed the problem because at least I could nudge it back in place. 3) GenomeLink - don't have C# api, so I tried my best to reroute the info into a JSON to send to my program. Never figured this out. 4) Sleep =.= zzz 5) Not finding each other earlier! We could've worked together and made something in Fove.

Anshul/ Karishma: We had never worked with Unity and VR, so getting even the basic stuff to work took time. Lack of Subject Matter Experts: Although someone spoke during the keynote about using technology to help kids with special needs, we could not connect to them during the rest of the event and so the internet was our source of help. We wanted to make sure we understand the kind of issues that children affected by Autism face. After building the proximity app, we were not sure if the other ideas were in line with our theme, so we spent a lot of time coming up with a feasible idea.

Accomplishments that I'm proud of

Helen: 1) I actually made something!!! I've never hacked before. 2) I learned Unity and C#. Now it'll be easier to start my next project. 3) I've started a project that I can actually use to help my friends with their autistic students. Karishma/Anshul: 1) We were able to identify a strong issue that kids with Autism face and were had a functional prototype for it 2) Glad could use technology for social good!

What we learned

1) Do not eat more than 4 AWAKE chocolates in 24 hours. The wrapping warns you. Don't do it. 2) The autism spectrum is much more drastic than we thought. I mostly designed around my friends' students, who are about moderately autistic. But there are so many more behaviors and problems than I ever knew. 3) Unity and C#. Especially exciting to think of all the games we can hopefully make :) 4) Programming on a computer with Unix is much easier.

What's next for Social Cues for Autism - Game/Data Analysis

Question Game: 1) Obtain a Tobii eye tracker from... somewhere. #broke 2) Figure out which parts of the face the user avoids and stares at. 3) Use GenomeLink to create more person specific game modes. 4) Collect and analyze data for autistic children to improve game. 5) Generate a comprehensive report for the teacher. 6) Make the game more visually appealing. 7) Try this out in classrooms.

VR Application: 1) Make the current application more appealing. We were aiming for a birthday party setup but couldn't even find a way to add balloons 2) The next thing we were trying to add was interaction. We were not sure how it should be incorporated and what kind of social interaction in VR was appropriate in case of Autism. Probably a meeting with a subject matter would solve this issue.

A comprehensive slideshow of our demo available here: http://bit.ly/2gBNLfF