Our project "Accessible AI" by Nooglers
Entered youtube URL, it transcribed and converted to Sign Language
Now comes another option , where you can enter text of Articles, books, etc....
The sign language video of text that you've provided
The final Option of uploading videos/audios from local directory without relying on Youtube
Corresponding Sign Language Generated for The video/audio which you've uploaded

AI for Change: Accessible AI - Enhancing Communication Accessibility

💡 Inspiration

Our team, Nooglers, was inspired by the communication barriers faced by the deaf and hard-of-hearing community in the digital age. We recognized the need for a tool that could bridge the gap between spoken language and Indian Sign Language (ISL), making digital content more accessible to this community.

🧠 What We Learned

Through this project, we gained valuable insights into:

The intricacies of Indian Sign Language and its grammatical structure
Advanced speech recognition technologies and their application
Natural Language Processing techniques for text parsing and manipulation
Video processing and synthesis for sign language representation
The importance of accessibility in technological advancements

🏗️ What We Built

Our web application offers a versatile platform for converting various forms of digital content into Indian Sign Language (ISL). Here's how it works:

Flexible Input Options: Users can provide input in multiple formats:
- YouTube URL
- Video or audio files from their local directory
- Text from articles or books
Automatic Transcription: For video and audio inputs, the application automatically extracts and transcribes the content using advanced speech recognition technology.
Text Processing: The transcribed or directly input text is processed and adapted to match ISL grammar structure.
Sign Language Mapping: The processed text is then mapped to corresponding ISL signs using our comprehensive dataset of sign language videos.
Handling Uncommon Words: If the system encounters words not present in the dataset, it dynamically combines individual alphabet signs to spell out the word.
Video Output: The final result is a cohesive video that visually represents the input content in Indian Sign Language.

This application bridges the communication gap between hearing individuals and the deaf or hard-of-hearing community, making a wide range of digital content accessible in ISL.

This is done in order to ensure that, We handle all types of Words as well as provide meaningful and accurate sign language video to Hearing Impaired

🛠️ How We Built It

Our "Accessible AI" web application was built using a combination of cutting-edge technologies and methodologies:

Speech-to-Text Conversion: We utilized Google's Speech-to-Text API for accurate transcription of spoken content from various input sources (YouTube URLs, local video/audio files).
Natural Language Processing: We employed the Stanford Parser for text parsing and implemented custom algorithms for tree manipulation to align the text with ISL grammar.
Text-to-ISL Conversion: We created a system that maps processed text to corresponding ISL signs using a comprehensive dataset of sign language videos and GIFs.
Video Processing: We developed a module to synthesize the final output video, integrating individual sign clips and non-manual signals.
User Interface: We used Streamlit to create an efficient and user-friendly interface for the application.

🧗 Challenges We Faced

Grammatical Differences: Adapting English grammar to ISL grammar posed a significant challenge, requiring careful consideration of sentence structures and word orders.
Dataset Creation: Building a comprehensive dataset of ISL signs and ensuring their accuracy was a time-consuming process.
Video Synthesis: Ensuring smooth transitions between individual sign clips and incorporating non-manual signals for natural-looking output was technically challenging.
Performance Optimization: Processing large video files and generating ISL videos in real-time required careful optimization of our algorithms and resource management.
Handling Edge Cases: Dealing with words or phrases without direct ISL equivalents required creative solutions, such as fingerspelling or contextual interpretation.

Despite these challenges, our team persevered, leveraging our collective skills and knowledge to create a functional and impactful solution. The "Accessible AI" project not only showcases the potential of integrating various technologies but also emphasizes the importance of accessibility in our increasingly digital world.

🏆 Accomplishments that we're proud of

Innovative Integration: We successfully integrated multiple complex technologies - speech recognition, natural language processing, and video synthesis - into a cohesive, functional system.
Accessibility Enhancement: Our application makes a significant stride in enhancing digital content accessibility for the deaf and hard-of-hearing community in India.
Flexible Input Handling: We developed a system capable of processing various input formats (YouTube URLs, local files, direct text), making it versatile and user-friendly.
ISL Grammar Adaptation: We implemented sophisticated NLP techniques to adapt English grammar to ISL grammar, ensuring more accurate and natural sign language output.
Dynamic Word Handling: Our system can handle unexpected words by dynamically combining individual alphabet signs, ensuring no content is left untranslated.
Efficient Video Processing: We optimized our video processing pipeline to handle the complexities of sign language synthesis, including non-manual signals and smooth transitions.
User-Friendly Interface: Using Streamlit, we created an intuitive and accessible interface that allows users to easily interact with our complex backend system.
Scalable Solution: Our application is designed to be scalable, allowing for future expansions of the sign language dataset and potential adaptation to other sign languages.
Cross-disciplinary Learning: Through this project, our team gained valuable insights across multiple domains, including linguistics, computer vision, and accessibility technologies.
Social Impact: We're proud to have created a tool that has the potential to make a real difference in people's lives, promoting inclusivity in digital communication.