HandPilot

• 

  Banner

• 

  Neutral Hand

• 

  Cursor Mode

• 

  Scroll Mode

• 

  Control Pannel

💭Inspiration💭

Accessibility of technology is one of society's grand issues that have yet to be resolved. Not everybody is given the same opportunity for education, healthcare, and employment. We acknowledge these discrepancies between people, and hope to minimize the gap for people that may struggle with "normal tasks". Our schools have programs for those with disabilities, and we can already observe how using technology is changing the lives for those that are not able to grab things easily, or do not have wide range of motion. As a common example, Yair's wrist had been broken too at one point, which made it impossible to grab things since he had a cast. HandPilot would address Yair's condition, and could solve a lot of problems that require fine motor skills going forward.

🤔What it does🤔

HandPilot converts hand movement to cursor movement, allowing for control of a computer (or any machine) without a mouse.

🔨How we built it🔨

• Brainstormed and decided to create a software program that would allow users to control the cursor on their computer screen using hand movements detected through a webcam.
• Used Python as their primary programming language and utilized libraries such as OpenCV, Mediapipe, Pyinput. - OpenCV for image processing, Mediapipe for perceptual computing to track the hands (uses machine learning), Pyinput to control the mouse.
• Designed a logo for HandPilot

💪Challenges we ran into💪

• Hand tracking did not work/was very choppy and buggy
• Invariable sensitivity was annoying to work with
• Some Python bugs here and there during development
• Running the .EXE did not work

🏆Accomplishments that we're proud of🏆

First, we can be proud that we came up with a new way to solve a common problem. We were creative and resourceful in finding a way to use a camera as a controller instead of a mouse or trackpad, which are more common ways to control a game.

Also, our project could affect how easy it is to get to places. The camera-based controller could be a new and easier way for people who have trouble using traditional input devices to interact with technology. This could help those people feel more independent and in charge of their lives.

We can also be proud of the technical skills that were needed to make this project happen. We may have had to learn and use skills in computer vision, machine learning, programming, and other areas. By putting these skills to good use and making a project that works, we have shown that we are very skilled in our chosen field.

Lastly, we can be proud of how our project could help people. If the camera-based controller is developed well and used by many people, it could change the way we interact with technology. Any group of developers can be proud of and happy about this kind of innovation and change.

🎓What we learned🎓

During the time we worked on this project, we learned a lot of important things. We probably learned more about computer vision and machine learning, as well as the challenges and complexities of adding a camera-based controller to software and systems that were already in place.

One thing we might have learned is how important it is for everyone in the group to talk to each other and work together. As with any project, there were probably different ideas and thoughts about where the camera-based controller should go and how it should be made. By talking to each other and working together well, the group could have made the development process go faster and avoided potential problems.

🔜What's next for HandPilot🔜

HandPilot, despite the name, will be extended to movement with other things, not just hands, to make it even more accessible. Also, more functions will be added, such as keyboard movement along with the current cursor control.

Built With

• ai
• mediapipe
• opencv
• python