Inspiration
Communication should not be a barrier for anyone. People who use sign language can face difficulties communicating with people who do not understand it, while people with speech or hearing difficulties may not always have access to suitable communication tools. We wanted to build a platform that makes these conversations easier by bringing sign language, text, speech, and video communication together in one place.
What it does
STS is a real-time communication platform designed to bridge communication gaps between sign-language users and speaking users. Its current features include: Gesture Recognition: Recognizes a limited set of supported hand gestures and displays their corresponding text. Speech-to-Text: Converts spoken words into captions using browser-supported speech recognition. Text-to-Speech: Reads written messages aloud. Real-Time Video Calling: Allows two participants to communicate through video and audio. Live Chat: Provides text messaging and quick communication phrases during calls. AI Sentence Refinement: Uses an NVIDIA-powered AI integration to help refine text. Room System: Allows users to create and join communication rooms. Authentication: Supports Google sign-in and guest access.
How we built it
We built STS as a web-based application with a separate frontend and backend. Frontend: HTML, CSS, and JavaScript, deployed on Vercel. Backend: Node.js and Express, deployed on Render. Real-Time Communication: WebRTC and Socket.IO for video-call media and signaling. Authentication: Firebase Authentication for Google sign-in and guest access. AI Integration: NVIDIA API for AI-powered sentence refinement. Speech Features: Browser-supported speech recognition and speech synthesis. The application combines these technologies into one communication experience rather than requiring users to switch between separate tools. Coding agent: calude and chatgpt
Challenges we ran into
One of our biggest challenges was integrating real-time video communication, gesture recognition, speech features, and chat into a single application. We also worked through frontend-backend communication issues during deployment, including configuring CORS between Vercel and Render. Making camera and microphone functionality work reliably across browsers and devices was another challenge.
Accomplishments that we're proud of
Built a working communication platform with real-time video calling. Integrated gesture recognition into the communication experience. Combined speech-to-text, text-to-speech, and live chat. Implemented room creation and joining. Integrated Firebase authentication. Deployed the frontend and backend separately. Created a publicly accessible live prototype.
What we learned
We learned how WebRTC and Socket.IO work together for real-time communication, how to integrate browser-based camera and microphone features, and how to deploy a full-stack application using separate hosting platforms. We also learned that accessibility-focused AI requires careful testing, reliable recognition, and honest communication about its limitations.
What's next for STS — Sign To Speech
Our next goal is to improve sign-language recognition by expanding the supported ASL vocabulary, improving recognition reliability, and exploring dynamic sign recognition. We also plan to improve the mobile experience, reduce recognition delays, and make communication more natural for both participants. Our long-term vision is to move toward more comprehensive sign-language interpretation while respecting the grammar and complexity of sign languages.
Log in or sign up for Devpost to join the conversation.