A lot of adults struggle with persistent feelings of self-doubt, which often stem from experiences in childhood. We also realized the potential of mixed reality to engage with a diverse audience all over the world; however, we also wanted to create something that was accessible. Since many people now have smartphones, we decided to create a therapeutic tool with augmented reality (AR) that is accessible across the world (in contrast to virtual reality that still has a high barrier of entry due to cost).
What it does
Pocket Grow is an AR mobile application that supports the expression of positive affirmation in children ages 5 to 9. This is a critical period in children’s identity development, according to a developmental psychologist, Eric Erickson. The application starts with a breathing exercise that puts the child in the mindset to begin thinking about positive affirmations. This is achieved by a virtual agent that is projected onto the physical space (i.e. augmented onto the physical space) and facilitates the breathing exercise. Children are then encouraged to say positive things about themselves, and the more things they say, the more flowers and other plants grow around them.
How we built it
Using Unity as the main engine, we utilized Google's ARCore SDK for Android devices in augmenting the virtual objects onto the physical space. For detecting and analyzing the user's speech, we used IBM Watson's Speech-to-Text, Tone Analyzer, and Text-To-Speech Unity SDKs. For quick prototyping, we used Procreate, Photoshop, Premier.
Challenges we ran into
None of us have used Unity or any sort of mixed reality/3D game development before. Additionally, we had originally planned to use Google's DialogFlow instead of IBM Watson, since we were already using their ARCore SDK. However, Google had deprecated and shut down their previous version 2 months ago, and there are no longer any Unity plugins for DialogFlow. We then had to shift gears and look for a new NLP tool, and since we had also not used IBM Watson with Unity before, this was also a learning curve. There were also no mentors that had experience with the technologies we wanted to use.
There was also limited documentation on everything given how new AR is.
Accomplishments that we're proud of
We are a very multidisciplinary team. We are majors in psychology, biology, art studio, and computer science. We are extremely proud to be able to come together and think of a product that would positively impact the mental health of children and their future. Although we did not get to achieve all our goals, we are proud that we could create a prototype in the short amount of time together.
What we learned
- How to use Unity
- Limitations of using new technologies such as AR
- Integrating different APIs into one project
- Communication across different disciplines
- How to come up with a minimum viable product
What's next for Pocket Grow
- Fully integrating the IBM Watson APIs
- Implementing the breathing exercises
- Use a wider variety of plants and having different species represent different goals
- Adding animals, because children love animals!