PolyLens

Inspiration

Over a billion people worldwide suffer from some form of hearing loss or deafness. In our increasingly connected yet isolating world, tools are needed more than ever to bridge the gap and encourage real human connections.

What it does

PolyLens detects sign language expressions from your webcam, models your hand internally as a series of 21 vectors each representing a 'critical point' on your hand, and outputs those normalized vectors. It produces high quality data that can be used to train machine learning models for any sign language worldwide.

How we built it

We used a React frontend with OpenCV and MediaPipe to read data from the camera and generate the corresponding vectors, which in turn were sent to our FastAPI backend (FastAPI chosen because, well, it's fast), to be processed and interpreted by our ASL model. A CSL (Chinese Sign Language) model is in development but training it was out of the scope of this hackathon. We also have a Firebase

Challenges we ran into

A lack of pre-trained models available and the short timeframe meant we were producing lots of high quality training data that we simply didn't have the time to consume. Further more, some of the most promising packages, libraries, and models for this specific application were outdated or outright deprecated.

Accomplishments that we're proud of

We're most proud of the data pipeline we established, connecting our React/JS frontend with our FastAPI/Python backend. This is an incredibly solid foundation that may be used in the future to build the next generation of sign language recognition models.

What we learned

Building real-time data pipelines. Applying computer vision libraries. Working efficiently and dividing labor in an effective manner. Containerization of the entire backend. Model training. Advanced dependency management, etc.