HandsFree

Hands Free! no need for a mouse or keyboard!

Comment 2

💡 Inspiration 💡

Have you ever been in a situation where you just didin't want or just straight up could not use you're keyboard and mouse? Rather just wished to control it with your mind? Well, we don't expect that to be common 🤣, but it is something we faced. HandsFree is an AI-based program created to help you navigate your computer with no physical touch at all!. Machines being able to register human movements and being able to interact with us has been our main source of inspiration for this project. We decided to implement a system where our own AI-powered app uses both Natural Language Proccesing and Computer Vision to recognize speech & gestures, then “thinks” and performs a command accordingly.

❓ What it does ❓

HandsFree is a desktop app that actively listens for keywords and gestures, then performs a command accordingly. It aims to completely minimize the need to use your mouse and keyboard.

🏗️ How we built it 🏗️

App:

  • App is built in Python, HTML, CSS, and JavaScript.
  • Mediapipe and OpenCV are used to recongize gestures.
  • SpeechRecignition is used to translate speech to text.

Website:

  • Website is built using React and Next.js.
  • Authentication System is built using Auth0.
  • Software Licensing is done using OpenScreen.

🚧 Challenges we ran into 🚧

  • Had trouble compiling our app to an executable file due to a variety of issues with the libaries we were using.
  • Had trouble getting the gesture recignition to work properely with dark lighting.

✅ Accomplishments that we're proud of ✅

  • Implemented a system that can recognize speech and gestures not just one or the other which has already been done before but a combination of both.
  • Implemented a creative use case of OpenScreen's API by creating a software licensing system.
  • Finishing this project in time, was by far the biggest accomplishment of this project.

🙋‍♂️ What we learned 🙋‍♂️

  • We learned how to use "eel", a Python libary that allows you to create a modern and customizable GUI using HTML, CSS and Javascript unlike tkinter and PyQt, while still having access to Python.
  • We learned how to use "OpenScreen" to create a software licensing system.
  • We learned how to use "Auth0" to create an authentication system.
  • We improved our knowledge on how to implement a classification system using "Mediapipe" and "OpenCV". ## 💭 What's next for HandsFree 💭 We plan on making this project more accurate by training the ML model extensively, to produce a smoother app. We also have plans to implement this program on mobile as well.

Built With

Share this project:

Updates