IllumiSign

• 

  Home Page

Inspiration

People with ALS (Amyotrophic Lateral Sclerosis) often lose their ability to speak and move due to nerve cell damage, making communication extremely difficult. Existing communication devices can be prohibitively expensive or inaccessible, leaving many patients without a reliable way to express themselves and no way of communication. Because of this, we wanted to create an easy to use open-source web-based communicator that empowers ALS patients to speak using only their eyes, without the need for specialized or costly hardware.

What it does

IllumiSign is a browser-based communication system designed for people with ALS. Using WebGazer.js, the platform allows users to select words or phrases simply by looking at tiles on a vocabulary grid for two to three seconds. Once the words are selected from the tiles, IllumiSign’s AI-powered Smart Compose feature can expand short selections like “thirsty” or “help” into natural, full sentences. These messages are then spoken aloud through the browser’s built-in text-to-speech engine. To maximize usability, the system also includes features like adjustable dwell time, customizable font size, high-contrast mode, voice selection, and a history replay function.

How we built it

IllumiSign was built entirely with JavaScript, HTML, and CSS. For gaze detection, we integrated WebGazer.js, which uses the webcam to track where the user is looking. We added smoothing functions and activation based on stationary eye placement to improve accuracy and reliability. Communication features were designed around emoji-labeled vocabulary tiles and local storage to save message history. To generate more natural speech, we connected the system to the Gemini API for Smart Compose, ensuring that user input could be quickly turned into fluent, expressive sentences. Finally, we used the browser’s Web Speech Synthesis API to provide real-time text-to-speech output, giving users an immediate voice.

Challenges we ran into

One of the biggest challenges was accuracy in eye tracking. WebGazer requires good calibration and proper lighting conditions, and making the system responsive enough for real use involved significant fine-tuning. Another challenge was balancing speed and accessibility. Stationary vision and scanning speeds had to be adjustable because every user has different physical capabilities. Integrating AI was also more complex than expected, as we needed to format Gemini’s output into clear, concise suggestions that users could select quickly. On top of that, we had to ensure the interface followed accessibility best practices, including ARIA roles, focus handling, and proper contrast for readability.

Accomplishments that we’re proud of

We are most proud of creating a fully functional communication system that runs entirely in the browser, requiring nothing more than a standard webcam. The interface which combines eye tracking makes IllumiSign adaptable for a wide range of users with different levels of mobility. We also successfully integrated AI Smart Compose, which helps users generate fluent, natural sentences with minimal input. Finally, we designed a modular vocabulary system that includes categories, emojis, and history replay, making communication both practical and personalized.

What we learned

Through this project, we learned how to integrate computer vision with accessibility and readability focused UI design and the importance of smoothing and calibration for reliable performance. We also discovered the value of user centered design in assistive technology, particularly the need for adjustable settings to accommodate different users. We also gained hands-on experience in applying AI prompt engineering in order to transform singular words into cohesive and understandable sentences, ensuring that generated suggestions were both helpful and easy to navigate. Most importantly, we learned that accessibility technology is not just about functionality but it's also about dignity and independence, giving people a way to be heard on their own terms.

What's Next for IllumiSign

Despite initial successes, the project definitely isn’t done yet. For one, the eye tracking mechanism can be improved for a higher success rate and accurate interpretation. Also, as of now, the tracking mechanism is confined in the word bank. For further developments, the eye tracking should be expanded to encompass the whole page for maximum user functionality. In addition, most ALS communication tech is too expensive for the general population of ALS-affected individuals. At its core, IllumiSign is meant to be a full-fledged, free interpreter for anyone to use, becoming the voice for those who cannot speak.

Built With

• css
• html
• javascript
• webgazer