Inspiration
Communication is essential in healthcare, especially during stressful or urgent situations. For Deaf and hard-of-hearing individuals who use American Sign Language (ASL), communicating with healthcare workers who do not know ASL can create significant barriers. I wanted to explore how technology could help bridge this gap and make medical communication more accessible.
That idea inspired EchoSign, a prototype designed to connect sign language and spoken language in both directions. My goal was to explore how AI could help people communicate their needs more easily in medical settings.
What it does
EchoSign is a prototype that explores two-way communication between ASL and spoken language through two main features:
- Sign to Speech: Uses camera input and AI-assisted sign recognition to identify supported signs and convert them into spoken phrases.
- Speech to Sign: Uses speech input and a collection of sign-language video clips and GIFs to display signs corresponding to supported phrases.
The application focuses on basic medical communication, such as asking for help, expressing needs, and communicating simple messages. Its current recognition capabilities are limited, and it is an early step toward making healthcare communication more accessible.
How I built it
I developed EchoSign independently with assistance from AI tools, including ChatGPT, Claude, and Google Gemini. I used these tools to help generate and refine code, troubleshoot errors, and develop the application's features and interface.
I used Python and Streamlit to build the application. The project incorporates Google Gemini for AI-assisted sign classification, speech recognition for spoken input, and ElevenLabs for text-to-speech output. I also connected supported phrases to sign-language video and GIF assets to enable the reverse communication workflow.
Throughout development, I worked on integrating these components into one application and refining the interface to make the main communication features clear and accessible.
Challenges I ran into
One of my biggest challenges was making sign recognition responsive and consistent. ASL signs can depend on hand movement, positioning, and context, which makes recognition more complex than identifying a single hand pose. AI processing latency also affected how quickly the application could respond.
Another challenge was connecting recognition results to the correct spoken phrases and matching supported phrases with the appropriate sign-language media. I learned that getting individual components to work does not automatically make the entire application reliable.
Because I developed the project independently with AI assistance, I also had to work through integration issues, troubleshoot errors, and refine the application as I discovered problems.
Accomplishments that I'm proud of
I'm proud of turning an idea into a working prototype that brings together multiple technologies for an accessibility-focused purpose. EchoSign combines a camera-based sign recognition workflow with speech output and a reverse workflow that connects spoken input to sign-language media.
I'm also proud of developing the project independently while using AI tools to explore unfamiliar technical challenges. Beyond building the interface, I worked to connect the different components into a single application centered on a real communication barrier.
Most importantly, EchoSign gave me an opportunity to explore how technology could help make healthcare communication more accessible.
What I learned
Building EchoSign taught me how to use AI-assisted development tools to help build an application, troubleshoot code, and connect different technologies. I gained experience working with Python, Streamlit, AI-assisted classification, speech recognition, text-to-speech, and multimedia assets.
I also learned that accessibility-focused technology requires more than an appealing interface. Reliability, response time, and accurate recognition are essential, particularly in healthcare settings. I gained a greater appreciation for the complexity of ASL and the importance of testing an AI system rather than assuming it can recognize every sign correctly.
What's next for Echo Sign
My next steps are to improve recognition accuracy, reduce response time, expand the collection of supported signs and phrases, and make the application more consistent when it cannot confidently recognize a sign.
I would also like to seek feedback from ASL users and accessibility experts to help guide future improvements. In the long term, I hope to explore how EchoSign could help reduce communication barriers between Deaf patients and healthcare workers.
EchoSign is an early-stage prototype, not a replacement for qualified interpreters or established emergency communication services. My goal is to continue improving it toward a more reliable and accessible communication tool.
Log in or sign up for Devpost to join the conversation.