Sign Vision AI

Draw anything, and our site figures out what it is—then gives it to you. No typing, just sketch and get results. It's the easiest, most fun way to search with your imagination!

Comment

Inspiration

We were inspired by the idea of making visual recognition more accessible and interactive. What if someone could draw something and instantly see what it is, visually and semantically? This could help bridge communication gaps for people with disabilities, support creative expression, and introduce kids to AI in a fun, intuitive way.

What it does

Sign Vision AI allows users to draw anything on a digital canvas. Once they finish drawing, our web app uses the MobileNet image classification model (via ml5.js) to guess what the drawing represents. It then displays the top three predictions with confidence levels, along with real-world images from Unsplash to help visually confirm the result.

How we built it

We used HTML, CSS, and vanilla JavaScript for the frontend interface. The ML backend relies on ml5.js, a wrapper for TensorFlow.js, and loads the pre-trained MobileNet model. We added features like pen thickness/color customization, touch support for mobile users, and dynamic image searching via the Unsplash API for enhanced visual feedback.

Challenges we ran into

Mobile compatibility and touch input took time to get right.

Getting reliable image classification on rough sketches was tricky due to the model being trained on real images, not drawings.

Integrating real-time Unsplash image search based on dynamic model output required careful filtering and formatting.

Accomplishments that we're proud of

Built a fully interactive and responsive sketch recognition interface.

Integrated AI-powered recognition and dynamic image search.

What we learned

How to use ml5.js and MobileNet for client-side image recognition.

How to handle canvas input, including drawing and resizing for model compatibility.

Real-time UI updates, error handling, and asynchronous JS patterns.

What's next for Sign Vision AI

Train a custom model based on sketches to increase accuracy.

Add voice support and multilingual labels for accessibility.

Allow users to save drawings and share results.

Turn this into a communication tool for users with speech impairments using sign/drawing input and visual + text output.

Share this project:

Updates