Spell Bound

Spell Bound is a VR learning game to help children with dyslexia and dysgraphia learn letter formation and word recognition.

Comment

Inspiration

The team formed with the goal of breaking down communication barriers between people through a tool is uses the strengths of VR. Children with dyslexia and dysgraphia often have frustrating experiences learning fundamental writing and reading skills, so we used the immersive nature of VR to engage their attention with an entertaining game while they learn.

What it does

Spell Bound is a VR learning game to help children with dyslexia and dysgraphia learn letter formation and word recognition. An animated wizard guides them through the process of writing letters with a wand to complete magic words and then reading the word out loud to cast the spell.

The app is intended to be used only under the supervision of a qualified professional such as an occupational therapist or special-ed teacher. These professionals use play activities for instruction and therapy, so we extended that methodology into an immersive, measurable learning tool.

The immersive nature of VR and the magical story elements keeps the child’s attention engaged while completing tasks that would otherwise frustrate them. Letter formation is most often taught in conjunction with handwriting, which is especially difficult for children with dyslexia/dysgraphia because of the fine motor skills handwriting requires. VR controllers rely primarily on arm movement for the interactions we designed, which allows the child to focus on letter formation separately from fine motor skills.

Performance is collected from gameplay into a dashboard which provides an assessment of the child’s development in improving their reading and writing skills. This information is used by the child’s instructor or reading/speech therapist to identify areas of improvement and determine the next course of action for instruction.

How I built it

We imported a wizard library into Unity for the environment and included an animated speaking mage who provides instruction to help the child get started with the game. At the start of gameplay, we provide a word with a missing letter. The mage communicates through an audio asset component which plays various .mp3 audio files depending on where the child is in their gameplay.

We needed to be able to detect the accuracy of the writing as well as provide instantaneous visual feedback, which is important for improving learning. When the want interacts with the letter outline spheres fade in to give a visual indicator. The score is measured by the distance from the center of the line.

We implemented hints throughout game plan depending on whether the child drew the requested letter correctly. We established a threshold to account for if the letter created with the wand is within a specified boundary. If the child draws a letter outside the boundary provided for the letter, they’re both given a hint (i.e. a brief look at the full letter or a trace outline) and asked to try again.

As the child correctly draws the letter, the size of the boundary decreases and the process of drawing the respective letter loops until the child draws the letter correctly. For the final level of gameplay, we used a method to invoke keywordRecognizer which listens for a keyword (i.e the full word that the child spells) and in return plays an audio asset which indicates whether the verbal pronunciation is correct or incorrect.

At the end of each session of gameplay, the results are sent to a dashboard for review by the therapist or teacher. The dashboard displays data and visualizations for the user’s proficiency per letter, average proficiency per session, and average proficiency over time. This information helps the evaluator to identify areas of struggle and track improvement.

Challenges I ran into

The biggest challenge was integrating the various methods we used such as collision detection to confirm the letters are drawn correct, speech recognition to check if the word is read correctly, audio narration to explain the game to the child, and many effects to keep the game engaging.

A surprising challenge with the writing was the wide variety of potential states such as the size of the letters, hints, and the various ways for the user pass and fail each stage. There were a lot of ways the game could react to the different states, so there were many unexpected outcomes along the way.

For the speech component, we realized that KeywordRecognizer makes it difficult to use conditional statements. Initially, we wanted to provide logic such as: if the user doesn’t say the keyword, then play “x” audio file. To overcome this, we used WaitForSeconds(5) to wait 5 seconds to check whether the user says the keyword. If the user didn’t say the keyword, then we played the ‘incorrect response’ audio asset. However, if the user said the correct keyword, then StopCoroutine(coroutine); stops the KeywordRecognizer to ensure that only the ‘correct response’ audio asset plays.

Accomplishments that I'm proud of

We dedicated a significant amount of time in the beginning towards planning which made a huge difference once we opened our laptops and began building! Although we had a plan in place, we were pretty agile in our process and were able to descope on the fly if it was determined that implementing a feature would be a drain on our resources (i.e. each other) and our time.

What I learned

There are various ways to get to the same solution - it just sometimes requires a bit of creative thinking! We iterated through 4 different methods of implementing speech recognition before settling on the solution we chose for this project. Also, since we are trying to do something we normally do in 2D software in a 3D environment. It’s not something people are often familiar with, so you can’t assume existing knowledge of interaction methods.

What's next for Spell Bound

Implementing additional challenges within the gameplay would be a great addition to Spell Bound. There are other dyslexia activities that could be translated well into a VR experience and it would be awesome to build those activities into the game.

Built With

Share this project:

Updates