Project VISION

Use only your head to fully control a computer.

Comment

Inspiration

After a video promoting ALS appeared in our feed, we became acutely aware of a demographic unable to use technology due deteriorating or deteriorated motor function. We decided we would build a project to help combat this.

What it does

Our program uses only a webcam to track a users eye movement and blinks. Simultaneously it is listening for any audio input. This way a person can demonstrate complete control of a computer by simply moving their head to move the cursor, blinking and keeping eyes shut to click and drag and talking to replace the use of a keyboard.

How we built it

We built our project using exclusively python. We separated into two groups. One group would take advantage of libraries such as openCL, media pipes and speech recognition, to build they eye tracker and input device as a whole. Whilst the other group would work on developing a demo, leveraging artificial intelligence and image recognition to fully promote the capabilities of our device in a unique and fun way.

Challenges we ran into

When it came to developing the input device we faced two unique challenges. The first was tracking eye movement in an efficient way, we constantly overloaded the computer leading to heavy lag. The second challenge we faced was actually developing an intuitive way to actually click and move. We were constantly adjust sensitivity of eye movement the time it takes to register a click and more until we created something that worked.

Accomplishments that we're proud of

We are very proud of creating a product that can actually help others, it provides accessibility and functionality that will ultimately improve people live. Best of all it is uncostly as the program works with a basic web cam. We couldn't be more happy about this.

What we learned

Our main take aways in completing this project are abundantly clear. We have learn how track not only head and eyes, but really any part of the body using openCL and mediapipes. Whereas on the demo development side of things we learnt of the difficulty that comes with integrating artificial intelligence into our work, just how tedious and demanding this can be.

What's next for Project VISION

Without there is more tinkering to be done. We could make our clicking system more intuitive, the camera could track our eyes better and be more stable too. However most importantly we could make the program more efficient, and therefore run more smoothly.

Built With

Share this project:

Updates

posted an update —

9:00am-11:52am - We completed our demo, uploaded it to youtube and devpost, got our drawing demo completed whilst uploading code to Github. It is nearing the end of the hackathon so we are handing everything in now.

The hackathon was a blast, thank you to all the organizers and donors for making this happen!

  • Team VISION

Log in or sign up for Devpost to join the conversation.

posted an update —

9am - We are aiming for a UI which is coming along well, but just in case we don't get it done in time we are making it terminal based but will still have the picture drawing feature. The picture drawing takes a screen shot and then clears it for the next round.

Log in or sign up for Devpost to join the conversation.

posted an update —

7:17am-8:02am - Image recognition and story writing artificial intellegence components individually completed now need to be put together. The screens for the story writing ai scenarios needed wrap around text to fit on the screen and look good. Beginning to finally put everything together to make completed demo.

Log in or sign up for Devpost to join the conversation.

posted an update —

Didn't realize that there was an option to give updates as we went so I will describe the general events that played up until now. From this point on I will give updates.

12:00pm-1:00pm - Early on we were researching what libraries we were going to implement in order to achieve the eye tracking we were looking for. Whilst creating a map of how our demo was going to progress.

1:00pm-5:00pm - We spent this mostly trying out different libraries and ideas to see what would be the best and most efficient way to track eye movement. On the demo side of things we were teaching ourselves the basics of implement artificial intelligence and graphic.

5:00pm-12:00am - This time was spent creating the individual aspects of input device. The blink to click features, the general eye movement sensitivity and cursor implementation, the audio to text feature. Whilst on the demo side of things we started to fine tune the artificial intelligence to write a story exactly as we want, and implemented the click and draw feature to demo.

12:00am-5:00am - This time was spent putting all the features of the input device together and getting them to run in order and parallel with the speech to text input. Whilst on the demo side of thing it was simply fine tuning and fixing errors brought about what was discussed previously.

5:00am-7:17am - This time was spent partially writing up the devpost document but mostly spent working on the demo, it is currently in partially completed fragments, that need to be completed and put together. It is looking like we will be in a time crunch.

Log in or sign up for Devpost to join the conversation.