FlickPoint

Inspiration## Inspiration

We came up with FlickPoint as a project based on computer vision, which many in our team were interested in. Furthermore, we wanted to tackle an interesting challenge of creating an accessible input device purely based on software.

What it does

FlickPoint is a program that uses hand gestures to control a computer. It also has speech to text for typing. This is very helpful for people with disabilities who may not be able to use their hands to control their device the standard way.

How we built it

For machine vision and hand tracking, we utilized the ML5 library, built on TensorFlow, in JavaScript. Using the coordinate data, we can identify the fingers' positions in relation to each other, and recognize the gesture being performed. In order to properly implement mouse movements, we built a Python API using FastAPI to work around the challenge of JavaScript being limited to its own tab. Finally, we originally created a script to utilize the ElevenLabs API to process Speech to Text inputs. However, we had to fall back to the much less feature-rich Webkit Speech Recognition to implement a stripped back version of this tool.

Challenges we ran into

Getting the client to communicate with our API was quite a challenge because of the frequency of requests being sent. This necessitated multiple rewrites of both client side and API code, even switching the API from Flask to FastAPI and working on a number of optimizations. On the Speech to Text front, our ElevenLabs API keys were disabled due to 'unusual activity' after we had already created a full script for it, so we had to make a last minute change to Webkit Speech Recognition, which required a major rewrite of code and a loss of multiple features.

Accomplishments that we're proud of

We are very proud of getting this ambitious project to a working state, especially with the combination of different fields that we needed to cover. The integration between these was often challenging, so it is a huge relief to see it through.

What we learned

We developed our skills greatly in various areas. Not only did we learn how to extract and analyze data from a machine vision model, two of our members have never used JavaScript before this hackathon, so this was a great learning experience for them.

What's next for FlickPoint

We would love to add virtual keyboard functionality on top of polishing our existing features for a more intuitive user experience, which after all is our end goal with this project.

Built With

• css
• fastapi
• html
• javascript
• ml5
• python
• tensorflow