DEAFINY

Searching for Classes
Calendar for Lessons
Log in menu
User Settings
Scheduled Lessons
Speech to text.
Classes

                                                       #DEAFINY#

Inspiration

Inclusive education is a fundamental right, yet many deaf students face challenges in traditional classrooms. We were inspired to create DEAFINY to bridge this gap and provide seamless communication for deaf students using sign language recognition. Our goal is to empower students and educators with technology that fosters equal learning opportunities.

What It Does?

DEAFINY uses advanced hand-tracking and sign language recognition to convert sign language gestures into text in real time and also converts speech to text/sign language to enhance education experience for students with disability. This allows deaf students to actively participate in classroom discussions, understand lectures, and communicate effectively with teachers and peers.

How We Built It?

We developed DEAFINY by integrating machine learning and computer vision technologies. For hand detection and tracking, we used the MediaPipe Hands SDK along with OpenCV.js to capture real-time hand landmarks. The mobile application was built with React Native and TypeScript, allowing it to run smoothly on both Android and iOS devices. To optimize performance, we implemented TensorFlow Lite Delegate and an H5-based neural network, making the model efficient for mobile deployment. Additionally, we utilized Firebase for cloud storage to use NoSQL, manage model updates. We used Firebase Authentication to store user data securely.

To further enhance our application, we worked with various debugging and optimization tools to improve efficiency. We resolved Expo and Gradle build issues by debugging AndroidManifest.xml errors and optimizing dependency configurations. Additionally, React Native Expo was used to streamline development, and we tackled real-time communication. Our work included extensive testing on different hardware configurations to ensure smooth performance across devices.

Challenges We Ran Into

One of the most significant challenges we encountered during the development process was implementing real-time image tracking and labeling using our model in a React Native environment. Achieving this required optimizing the camera input, ensuring efficient hand landmark detection, and integrating a lightweight classifier capable of running smoothly on mobile devices. Additionally, handling the real-time processing pipeline while maintaining performance and accuracy posed technical difficulties, as we needed to balance computational efficiency with the responsiveness of the application.

Aside from real-time image tracking and labeling, another major challenge we faced was implementing a real-time speech-to-text and sign language conversion functionality in React Native. This required integrating a highly efficient speech recognition engine capable of accurately transcribing spoken words into text while ensuring minimal latency. Additionally, mapping the transcribed text to corresponding sign language gestures presented further complexities, as it involved synchronizing the sign recognition model with real-time audio processing. Optimizing this functionality for mobile devices while maintaining smooth performance and accuracy was one of the most technically demanding aspects of our development process.

Accomplishments That We're Proud Of

We successfully implemented real-time sign language recognition on mobile devices, allowing for seamless communication through gesture detection. By optimizing the model, we ensured that it runs efficiently even on low-power devices, making it accessible to a wider audience.

Creating an inclusive educational tool that can positively impact deaf students was a significant achievement, as it promotes equal learning opportunities. Additionally, we overcame various technical challenges to provide a smooth and reliable user experience, ensuring that the application is both functional and user-friendly.

Moreover, we successfully resolved Expo build issues, improved network request handling for image uploads, and optimized the React Native environment for mobile AI deployment. Our ability to debug, refine, and enhance the application through iterative testing was a major milestone in the project.

What We Learned

Through this project, we gained a deeper understanding of the importance of accessibility and inclusion in education, recognizing how technology can help bridge communication gaps for deaf students. We also learned valuable optimization techniques for deploying AI models on mobile devices, ensuring efficient performance without compromising accuracy. Real-world testing proved to be essential in improving model performance, as it allowed us to identify and address practical challenges. Most importantly, this experience reinforced our belief in the potential of AI to break communication barriers and create a more inclusive world.

Additionally, we learned best practices for debugging Expo-based React Native projects, handling Google Play deployment issues, and managing Firebase authentication and cloud storage effectively. We also explored alternative video call systems, leading to a better understanding of real-time communication solutions beyond WebRTC.

What's next for DEAFINY

Moving forward, we plan to expand DEAFINY’s support for multiple sign languages, making it accessible to a more diverse user base. We also aim to enhance gesture recognition accuracy by training our model with a larger and more diverse dataset. To further improve communication between deaf and non-signing individuals, we intend to integrate voice synthesis, allowing recognized gestures to be converted into spoken language. Additionally, we will develop a web version to increase accessibility beyond mobile devices. Lastly, we hope to partner with educational institutions to implement DEAFINY in real-world classrooms, ensuring that deaf students can fully engage in their learning environments.

Furthermore, we plan to refine React Native Expo configurations, and explore alternative AI model deployment methods to make the system more scalable and responsive.