Inspiration
Our inspiration stemmed from the desire to empower the deaf and hard of hearing community by providing them with a more inclusive means of communication. We recognized the importance of American Sign Language (ASL) as a primary mode of communication for many individuals and sought to leverage technology to make ASL more accessible in virtual environments.
What it does
ASL GestureSense Unity is a groundbreaking project that enables real-time recognition and interpretation of American Sign Language (ASL) gestures in virtual environments. It allows users to interact naturally and intuitively with digital applications using ASL, bridging communication gaps and fostering inclusivity.
How we built it
ASL GestureSense Unity was developed using Unity, Met Quest 2, and various XR interaction toolkits. Unity provided the foundational environment for our project, allowing us to leverage its powerful rendering capabilities and cross-platform compatibility.
The integration of Met Quest 2 extended our project's capabilities with its advanced hardware features, including high-resolution displays, precise motion tracking, and hand gesture recognition. We harnessed these features to create immersive experiences that closely mimic real-world interactions.
In conjunction with Unity and Met Quest 2, we employed XR interaction toolkits such as the XR Interaction Toolkit and VRTK (Virtual Reality Toolkit). These toolkits enabled us to implement complex interaction mechanics, including hand tracking, gesture recognition, and object manipulation. Additionally, VRTK complemented our development efforts by offering a wide range of tools and utilities designed to streamline the creation of interactive VR applications. With VRTK, we were able to enhance our project with advanced hand gesture recognition capabilities, enabling precise detection and interpretation of gestures performed by users' left and right hands.
We utilized Unity's native scripting API along with custom shaders and physics simulations to achieve lifelike interactions and visual effects.
Throughout the development cycle, we conducted rigorous testing and optimization to ensure optimal performance across different devices and platforms.
Challenges we ran into
During our journey, we encountered a variety of challenges that tested our problem-solving skills and resilience. One significant hurdle was grappling with compatibility issues between Meta Quest 2 and both mobile phones and laptops. Ensuring seamless interaction across different devices proved to be a daunting task.
Furthermore, we faced numerous build issues, particularly in configuring project settings and managing packages. Unity editor version compatibility also emerged as a persistent issue, requiring careful navigation and troubleshooting to maintain project stability and functionality.
Moreover, addressing boundary loss in Meta Quest 2 controllers presented its own set of challenges, demanding innovative solutions to ensure reliable tracking and user experience in virtual environments.
Despite these obstacles, our team remained dedicated and resourceful, leveraging our collective expertise to overcome each challenge and propel the project forward.
Accomplishments that we're proud of
One of our proudest achievements is venturing into virtual reality (VR) and Meta Quest 2 for the first time. Despite the novelty and complexity, we swiftly integrated the hardware with our software in a remarkably short period. Navigating VR hardware and establishing seamless communication demanded dedication and perseverance, expanding our technical prowess.
Witnessing our user interface (UI) on the Meta Quest platform was a significant milestone. For VR novices, this integration validated our adaptability and enthusiasm. It fueled further exploration in VR design and development, marking a transformative journey into uncharted technological terrain.
What we learned
Our project journey underscored the value of embracing new challenges without hesitation. We learned that perseverance and a willingness to explore the unknown are vital for growth. By confronting difficulties head-on, we gained invaluable insights and experiences that transcend technical skills. Above all, we discovered that pushing beyond our comfort zones fosters resilience and adaptability, essential qualities in the dynamic world of technology.
What's next for ASL GestureSense Unity
Our vision extends beyond gesture recognition. We aim to incorporate subtitle features for hand gestures, enhancing accessibility and learning opportunities. Additionally, we're committed to developing comprehensive tutorials to facilitate the learning of sign language. By combining innovation with education, we aspire to empower individuals and promote inclusivity on a broader scale.
Log in or sign up for Devpost to join the conversation.