Our Inspiration

The last two years have shown that there are very limited options for interactive remote learning tools. This issue has an impact on both standard education and diagnostic education, especially in children. Both the AWS and the Children's Health challenges addressed these issues.

Children's Health sought out technological solutions that would make the diagnostic, treatment, and recovery process more engaging and interactive. Amazon issued a challenge to create a solution that decreases the digital divide in remote education, as well as promotes diversity and inclusion in the process.

Our team believed that these challenges could go hand in hand, resulting in a solution that could provide accessible, remote education in an exciting and meaningful way for children. After talking with the Children's Health representative, we decided that an application that catered to a child's health with engaging games, education activities, and diagnoses explanations would be a great way to tackle these issues.

What it does

ARRTIE (Augmented Reality Remote Teaching Interactive Experience) is an android/ios app that allows the user to engage in educational activities using AR. The final version that we made was catered to a hypothetical child that had been diagnosed with Type II Diabetes and Obesity. The application provides information about each diagnosis, as well as two interactive AR games that can help the child form good nutritional habits to maintain their health from the palm of their hand.

The first game was designed to help the child identify foods that would and would not help them maintain good nutritional health. Example foods were listed in a legend on the 'game name' menu screen. In-game, these foods were attached to balloons in AR space, and the child was asked to pop/ get rid of the foods that would not help their nutritional health (ex: chips, candy, soda.). The student would then obtain points based on how many balloons they could identify and pop correctly.

The second game was designed to teach the child about the importance of balance in food intake. Maintaining a good balance of carbohydrates, lipids, and proteins can help decrease the intensity of the effects of type II diabetes. 'game name' has the child pop balloons of foods in each of the three main biomolecule groups: carbohydrates, lipids, and proteins. By popping these balloons, they rack up points for each category in a bar. Their goal is to fill up the bar just right, not to under or overfill it.

The third feature that we implemented into our application was the Education Displays feature. The goal was to allow a teacher and their student, or a doctor and their patient, to interact remotely using AR. The teacher/doctor would be able to pull up a diagram or image in 3D AR space that the student could see on their phone, they could then converse or discuss the topics, while also jointly interacting with the diagram. For example, if the doctor asked their patient to indicate which part of MyPlate a certain food would fall into, the student could draw a circle on the diagram, and that circle would show up on the teacher's end as well. This would be done with AWS and gamelift to provide a multi-user capability.

Overall, this app was designed to give children a more interactive and exciting resource to learn and grow, while also providing access to unique and specialized learning tools.

How we built it

After our initial brainstorming process, and decision of features, our team planned out the application in a flow chart, designating areas of focus, and allocating tasks to each member.

The main areas of focus were: User Interface (Menus, information), AR game(s), Education Display (incorporating AWS, and multiuser capabilities.)

For the user interface, we utilized Unity's scenes and scripting to navigate through four different menus: the main menu, a description of the diagnoses, and two different simple instructions for our games. We kept things brief and playful in order to better appeal to children.

In addition, we wanted to use original art for the backgrounds, logos, and icons. All of the drawings seen in the background of the UI menus, as well as the icons on the balloons, were drawn by our team members!

The AR games were designed in unity, incorporating AR SDK (specify SDK). (neil/ abhishek fill in)

The Education display was created through (neil/ Phillip/ erin fill in)

We also frequently tested each version of the app in order to catch bugs and errors.

Challenges we ran into

A big challenge our team struggled with was understanding and implementing AWS and gamelift for the multiuser features. (can either abhi or neil elaborate)

Another issue was displaying our 2D images correctly on 3D objects, we originally tried to create multiple textures and materials to solve the warped look this caused, but eventually found a way to have the image attach to an invisible plane right in front of the objects,

Game development issues (bugs errors, can neil or abhishek elaborate)

Accomplishments that we're proud of

Our team worked extremely hard over the past few days and nights, we had a huge ambition to create a working AR application. There are a few pieces of our project that we are especially proud of. For one, getting AWS set up and successfully integrated into our project was a huge accomplishment.

We were also greatly satisfied with our fully functioning UI menu, complete with working scenes, buttons, and original art for each icon, logo, and background piece.

Another accomplishment that we are proud of is our success in creating three distinct features within the app: The Diagnoses section, Game options, and the Education Display.

What we learned

This project was actually many of our team member's first time using Unity. 3/4 of us had little experience with coding in Unity, much less AR programming within it. Yet by the end, we had created a fully functional AR app using the Unity platform!

We also learned how to set up an AWS server and integrate a Unity project into it.

What's next for ARRTIE

We believe that ARRTIE could be used to encourage children to understand their diagnoses and to apply their knowledge. Although we only catered to a hypothetical child with two diagnoses for this version of the app, we believe that the app could easily be configured to fit many other diagnoses.

In addition, we believe that the features could be used for standard education as well (especially in a remote setting). We think a great additional feature would be to allow teachers and doctors to configure the games on their end to meet their specific student's/ patient's needs,

We would also like to implement levels and difficulty so that with more use of the app's features, the more difficult and challenging the games can become. It would also be engaging to have the games have location-based levels, meaning if a patient or student walked to a certain area they would be able to play a game specific to that area (encouragement of movement with learning).

Finally, we believe that expanding the Education Displays to include a group whiteboard or group drawings could also encourage engagement with remote learning.

Share this project:

Updates