Inspiration

Millions of people who use Indian Sign Language (ISL) face communication barriers while accessing essential urban public services such as hospitals, banks, government offices, police stations, railway stations, and educational institutions. Existing sign language systems often recognize only isolated gestures, provide English-only output, or require expensive hardware. We wanted to build an affordable, AI-powered communication platform that enables seamless multilingual communication using only a standard laptop webcam, making smart cities more inclusive and accessible.

What it does

Our system uses a laptop webcam to recognize both-hand Indian Sign Language gestures in real time and interpret them at the sentence level. The recognized text is translated into multiple Indian languages, including English, Hindi, Tamil, Malayalam, Telugu, and Kannada, and converted into natural speech. This enables effective communication between ISL users and public service providers without requiring interpreters or specialized hardware.

How we built it

The proposed workflow combines MediaPipe for real-time hand landmark extraction, a Transformer-based pose-to-text recognition model for sentence-level ISL recognition, IndicTrans2 for multilingual translation, and a Text-to-Speech engine for voice generation. Our system architecture is based on recent research published between 2024 and 2026, ensuring that the design follows current advances in computer vision, deep learning, and multilingual language technologies.

Challenges we ran into

Continuous sentence-level ISL recognition remains an open research problem due to limited datasets and complex gesture sequences. Achieving accurate recognition while maintaining real-time performance using only a standard webcam required balancing model accuracy with computational efficiency. Another challenge was designing a multilingual communication pipeline suitable for practical deployment in urban public environments.

What we learned

This project helped us understand that accessibility is not only a social issue but also a technological challenge. Through recent research, we learned that most existing systems focus on isolated gesture recognition or single-language output. Integrating sentence-level recognition, multilingual translation, and speech generation into one practical platform offers a stronger solution for real-world communication.

What's next

We plan to expand the ISL sentence vocabulary, improve recognition accuracy using larger datasets, support additional Indian languages, develop a mobile application, introduce offline inference, and extend the platform with two-way communication (speech-to-sign). Our long-term vision is to deploy this solution across smart urban public services to improve accessibility for the Deaf and hard-of-hearing community.

Built With

• accessibility
• ai
• computer-vision
• coqui-tts
• deep-learning
• indian-sign-language
• indictrans2
• machine-learning
• mediapipe
• multilingual
• opencv
• python
• pytorch
• sign-language
• smart-city
• streamlit
• transformers