FlappyLingo

Inspiration

As of the recent pandemic, interacting and communicating with marginalized communities has become even more difficult. This is especially true for those who are hearing-impaired, as they primarily communicate through lipreading (masks are a barrier) and writing (social distancing guidelines & risk of spreading germs or the virus). Therefore, many deaf/hard of hearing people at hospitals/other institutions are unable to consistently communicate with doctors/staff. Even ASL interpreters are unable to be there in-person as a result of covid-19 restrictions, and remote interpreters often face audio, camera and other technical issues, making it extremely stressful for those hard of hearing. That is why we have created FlappyLingo, a fun, interactive application for people of all ages to learn American Sign Language (ASL) and communicate with those who are hearing-impaired. Helping the future generation learn about and appreciate those with a hearing disability, making it easier for deaf people to communicate with those around them.

What it does

Similar to Duolingo, it teaches users a new language (ASL) but with the addition of flappy bird. Users must demonstrate the correct hand sign to pass each flappy bird pillar, if not the flappy bird will fall, lifelessly to the ground, forcing users to restart. This makes the learning process much more enjoyable and interactive.

How we built it

Mediapipe API to draw hand skeleton and recognize gestures
Pygame to create the application interface and flappy bird
Sklearn to import the K-Nearest Neighbour (K-NN) model to train the sign language model
Pickle to compile custom made data

Challenges we ran into

Errors when syncing gestures to the flappy bird (pygame failed to categorize)
Pickle files often corrupted and would not compile the code properly (all pictures must be retaken)
K-NN model would not import properly
Flappy bird would fly out of screen due to poorly written screen constraints
Not enough sign language phrases