Inspiration
Educators are looking for ways to convey information that is meaningful, accessible, and promotes inclusion. Teaching students with disabilities can be challenging. Augmented Reality (AR) is a type of technology that adds digital information to the physical world. It involves the overlay of computer-generated graphics, sounds, and other virtual elements onto the real world, creating a blended and interactive experience. It can be accessed through various devices such as smartphones, tablets, head-mounted displays, and smart glasses. When it comes to education, AR has the potential to act as 1) assistive technology enabling sensory substitution and 2) a platform to display information in a way that involves more senses, is contextualized, and promotes interactivity and engagement. The integration of this technology can seamlessly elevate the quality of the material delivery by providing a spatialized multimodal learning experience to students, facilitating the learning process while giving educators more agency over their practices.
Example Use Case Story
John is a biology teacher who wants to bring change to his classroom. He wants to integrate the MirageXR app into his teaching methods as a way to cut costs while providing interactive ways to display visual information to stimulate interest and prompt sense-making. John also teaches deaf and hard of hearing (DHH) students and wants this activity to be accessible to them. Although John is trained in American Sign Language (ASL), teaching using ASL while recording content on MirageXR can be challenging and becomes next to impossible when it comes to giving homework or any remote activity.
What it does
The current project aims at combining assistive technologies and new richer teaching practices through AR to empower both educators and learners with disabilities, in this case, DHH learners. MirageXR as a mobile-based AR solution will be enhanced with several new features to 1) enhance the learning experience of students by providing an alternative embodied display of the material, 2) bolstering the skillset of the educator, and most importantly 3) develop inclusivity of the classroom environment by enabling access to the material to DHH learners. Objective 1) will be reached by leveraging the dynamic potential of AR to add visual elements on top of key information to increase visual saliency and attention focus by dynamically highlighting (part of) relevant 3D objects. Objective 2) will attain itself through the use of the tool by educators with aim 3) will be achieved using a new voice recognition-based ASL feature that will convert speech inputs into ASL gestures displayed on the phone as 3D models.
How we built it
To implement this idea, we decided to use Unity 3D Game Engine as the main development platform. Unity was chosen because the MirageXR tool is built in Unity. To successfully create a prototype, we would need to add the following 2 additional features to MirageXR - Voice Recognition 3D models for ASL
The tech resources required to implement this feature are listed below -
AR Development Engine - Unity MirageXR is developed in Unity Speech Recognition - Microsoft Azure Speech SDK 3D Modelling - Blender/sketchfab.com
Our plan was to develop a voice recognition-based system using Microsoft Azure Speech SDK and ASL 3D models from sketchfab.com. This feature could be added to both smartphones and head-mounted devices such as Microsoft Hololens2. We planned on developing a C# script which would translate the speech detected by MS Azure Speech SDK to 3D models of ASL, and would display the relevant animations in a corner of the screen or view the activity in MirageXR. This would allow the end-user (students) to see the AR animation in 3D along with ASL signs in the corner of their view.
There is a plethora of documentation provided by Microsoft here on how to integrate Azure Cognitive Services into Unity. The developers on the team had experience in using Azure Speech SDK in C# to convert voice to text and then use the voice-generated text to perform related ASL signs in Unity.
Challenges we ran into
Unfortunately, we were not successful in importing the MirageXR development branch from Github to Unity. The reasons were the time constraints of the hackathon and certain Vuforia-related errors which can be seen in the screenshot below. Although the Unity version on developers’ PCs had the latest versions of Vuforia on them, we could not get rid of these errors. Due to time constraints, we could not investigate this issue any further and have decided to submit our report with a code. Nevertheless, the dependency of MirageXR on Unity ensures that voice-recognition-based features can be integrated into the tool seamlessly. Moreover, use of C# scripting in Unity ensures easy conversion and manipulation of 3D models and animations through voice-based input.
Accomplishments that we're proud of
We have proposed a voice recognition-based Speech-to-Sign Language translator using Unity, Microsoft Azure Speech SDK and C# scripting. We believe it can be seamlessly integrated into MirageXR as it is developed in Unity.
What's next for Voice Recognition-based Sign Language feature in MirageXR
As an assistive technology in educational ecosystems, the integration of audio-visual features as a way to enable sensory substitution is not only a classroom tool but a medium that has a potentially much wider societal impact by changing the way the teaching material can be appraised and assessed. By promoting inclusivity and negating the disabling factors of the teaching environments for students with DHH, the upgraded MirageXR can change the ways these students are schooled - thus redirecting public funding to more important resources and cutting the costs related to the need to consult experts. Because the application can be deployed on many portable displays, it is bound to withstand future changes without being obsolete. The approach is extremely straightforward and it is easy to deploy on mobile phones and tablets in the classroom.
Please note - we submitted the link to WEKIT ECS - Mirage XR video on YouTube as we have not created one for our submission.
Log in or sign up for Devpost to join the conversation.