SignSpeak

• 

  pic

• 

  pic

• 

  pic

Inspiration

Communication is a fundamental human right, yet many people who are deaf or non-verbal face significant barriers in interacting with technology and others. While voice assistants and conversational AI have become mainstream, they often exclude those who rely on sign language. I was inspired to create an accessible bridge that empowers the deaf community by translating American Sign Language (ASL) in real time.

What it does

SignSpeak uses AWS DeepLens to capture and process live video of ASL hand gestures, recognizing individual letters of the ASL alphabet. Leveraging a deep learning model trained on a custom dataset, it translates these signs into written and spoken English in real time, enabling seamless communication with computers and other users. I plan to enhance it further by integrating AWS generative AI services to convert recognized letters into meaningful sentences and generate conversational responses.

How I built it

I developed a custom dataset of ASL alphabet images, focusing on static hand signs for reliability. Using Amazon SageMaker, I trained a vision model based on transfer learning with SqueezeNet. The model runs on AWS DeepLens, with inference results streamed through AWS Lambda and IoT services. The frontend interface displays recognized letters and builds sentences live. The architecture prioritizes edge computing for low latency and privacy.

Challenges I ran into

Recognizing dynamic ASL letters like 'j' and 'z' was challenging due to motion complexity, so I focused on static signs for initial accuracy. Training a robust model with limited labeled data required careful dataset creation and augmentation. Ensuring real-time performance on DeepLens edge hardware while maintaining accuracy was also demanding.

Accomplishments that I'm proud of

I successfully built a working ASL recognition system that translates signs into text with high accuracy and real-time feedback. The custom dataset and training pipeline demonstrate how edge AI devices can empower accessibility. I also architected a scalable, cloud-integrated solution combining AWS services for streaming, storage, and deployment.

What I learned

This project deepened my understanding of AWS AI/ML services, edge computing, and IoT integration. I gained hands-on experience with transfer learning for computer vision, real-time inference on constrained devices, and building end-to-end AWS pipelines. Most importantly, I learned how technology can be a powerful tool for social good.

What's next for SignSpeak

Expanding support for dynamic ASL signs and other sign languages is a key priority. I also plan to improve the UI for greater accessibility and explore partnerships to bring this technology to communities in need.

Built With

• aws-deeplens
• cloudformation
• cognito
• docker
• iot-core
• lambda
• mqtt
• python
• s3
• sagemaker