SignSight

Inspiration The idea for SignSight was born out of the desire to create a more intuitive and touch-free way to interact with digital devices. With the rise of gesture-based controls in modern tech and the growing emphasis on accessibility and hygiene, we envisioned a system where users could control their monitors with simple hand signs—no mouse, no keyboard, just gestures. This not only enhances the user experience but also opens doors for inclusive technologies for individuals with limited mobility.

What it does SignSight is a web-based application that uses your webcam to detect and interpret hand gestures in real time. By recognizing predefined hand signs—like open palm, fist, thumbs up, and swipes—it enables users to perform common monitor control actions such as scrolling, tab switching, adjusting volume, and navigating the web, all without touching their computer. The interface is intuitive, responsive, and designed to work entirely in the browser.

How we built it We used HTML, CSS, and JavaScript to build the frontend, ensuring a clean and futuristic user interface. For gesture recognition, we integrated MediaPipe Hands and TensorFlow.js, which allowed us to capture hand landmarks and classify gestures in real time using the webcam. We implemented responsive design principles to make the site mobile-friendly and added smooth CSS animations to enhance visual feedback. Gesture mappings were handled in JavaScript, enabling real-time interaction without needing a backend.

Challenges we ran into Real-time hand tracking and gesture classification came with performance issues, especially on lower-end devices. Ensuring accuracy while reducing false positives was tricky. Handling multiple lighting conditions and keeping the UI lightweight yet visually rich also required careful design and testing. Integrating gesture detection into seamless browser interactions while avoiding accidental triggers posed a unique challenge.

Accomplishments that we're proud of We’re proud of building a fully functional AI-powered gesture control system that runs entirely in the browser. The project merges real-time computer vision with intuitive UI/UX design, all packed into a single-page application. Achieving smooth gesture recognition without relying on external servers or complex installations is a highlight of our solution.

What we learned We gained deeper insights into working with real-time video feeds, hand landmark detection, and TensorFlow.js. We also learned how to optimize JavaScript code for real-time applications and how crucial UX design is when building interfaces for alternative input systems like hand gestures.

What’s next for SignSight Moving forward, we plan to expand the gesture set and add user customization for gesture-to-action mapping. We also aim to integrate voice and gesture combinations for even more powerful control. Further improvements could include accessibility modes, visual calibration, support for smart TVs, and deploying as a browser extension for enhanced compatibility across websites.

Built With

• https://shimmering-empanada-3ea18c.netlify.app/