SASL in AR

An Augmented Reality Resource that utilises Hand Tracking to teach the SASL Alphabet, while implementing interactive elements of gamification and collecting users' analytic data through social media.

Comment

Inspiration

SparkAR Studio (the Augmented Reality engine for Instagram and Facebook) recently allowed for hand tracking effects, and the bulk of effects I have seen created were mirrored on either hand. It would be a cloud or a fireball or some 3D object that would be mirrored on each hand. I saw the opportunity to split each hands functionality. If you have 2 hands, you can allow for 2 different functions. South African Sign language is my 3rd language, and all apps or related educational materials seem to just be a collection of videos. I've always aimed to make my Augmented reality work more engaging and interactive, so this seemed like the perfect moment to develop new materials and see how it translates on a digital platform. I also wanted to use the social media platform infrastructure as a way to promote and educate others of a regularly marginalised demographic and language.

What it does

A separate animation sequence runs on each hand, you hold up your left them right hand and you see after a "ready" statement and a 3 second countdown the sequence runs through the alphabet, on the opposite hand a semi-opaque image of the correlating hand shape appears. Prompting the user to mimic the hand shape. The sequence runs at 1 frame per second, with each individual frame repeated. So each hand shape and letter is visible 2 seconds at a time. Once you reach the end of the alphabet it cycles through the sequence again. It was created for users to share their knowledge of language on social media, and because younger generations have grown up to accept social media platforms and the UI, there is no need for user education or to spend money and time to develop a new app where the analytics aren't as accurate.

How we built it

I split the hand tracking functionality on the SparkAR Hand tracking template. Drew up the alphabet and hand shapes, ran a separate animation sequence on each anchored hand, made sure that they synced at over 60 frames per sequence. Made sure that the canvases on each hand scale to always cover the hand and start at the wrist for the most part (open or closed, close up or further away), allowing the user to always accurately copy the hand shape.

Challenges we ran into

I had to learn hand tracking and the requirements to make a useful effect. It's very easy to create an effect that utilises social media capabilities but is out of the size requirements. Luckily I've had to compress effects in the past. And after user testing with some friends, I got feedback that made me double up the frames of the sequences because prior to testing, users had only 1 second to copy the hand shape, instead of the now 2 seconds. Which came from the fact that I've known the alphabet for years now, and took for granted the learning curve required to learn a new language. Also, in a business sense I always prepare a demo first before approaching clients (including a project like this that I can take to my school clients), which means I have to allocate time and money to development before a client is confirmed. It is beneficial when explaining AR, but that I need to seek out clients willing to accept my costing and processes otherwise I do end up demo'ing for months on end at my expense. Fortunately once a client is on board with the concept they are willing to pay for licensing and continual development seeing the benefits of driving traffic to their social media profiles and the work they do.

Accomplishments that we're proud of

About 4 years ago I decided to learn game design, I taught myself C#, Unity, SparkAR, music editing, graphic design and started my own game design company. I published games on The Apple app store (where my one game made it to the number 1 spot in its category) as well as the Google Playstore. I won MTN app of the year best gaming solution 2020 and was nominated for 2 Huawei innovation awards in the same year. I got accepted into a development programme in the UK and now get to explore making educational games for schools and other educational institutions. My Instagram Augmented Reality filter profile currently has 70 million impressions, which is all insane for a self-taught theatre major using a 2012 Macbook.

What we learned

Hand tracking and the processes and coding involved.

What's next for SASL in AR

The concept and demo files were sent to Inclusivity Africa's development programme where accepted participants are announced later this year. I've been accepted into a social innovation programme where some participants are have created training programmes for the deaf. I get to test with their learners to develop more refined materials on a larger scale. And since now I have a proof of concept demo (as well as the backend for hand tracking) working I can develop 2 handed signs as well as scale to accommodate all non-verbal signed languages. I have the added benefit to be developing in a digital space, so material costs are close to zero, making it cheaper and quicker to scale globally and find an international clientele.

Built With

  • analytics
  • ar
  • data
  • handtracking
  • instagram
  • mobile
  • sparkarstudio
Share this project:

Updates