We set out with the goal of being able to automate the tedious tasks that one does on a day-to-day basis. Really just the simple things that can just be done a lot more conveniently. This meant opening web pages, applications etc. Gradually, we realized that we had stumbled upon a great way to enable the differently abled to access computers. At this point, we began to prioritize the ease of use on part of the user. By incorporating an OCR, we were able to locate (and click) elements like buttons that traditional web page analyzers cannot. By the end, We had extended to the ability to control computer peripherals beyond just a mouse to the keyboard as well. To top it all off, we created a variety of "voice-triggerable" scripts that make computer usage even more of a breeze!
What it does
It is a voice control tool that allows the user to control their computer.
How we built it
Everything was programmed in Python. A simple script was used to handle mouse and keyboard controls. The GUI was created using Tkinter. The speech to text was generated using Google Cloud. The text to speech was generated with MacOS's built in terminal commands. The mouse tracking works by having the user say the text on the screen at which to move the cursor. The text is then search for by taking a screenshot and then running an OCR (with Tesseract) to find the exact locations.
Challenges we ran into
- trouble with installing libraries
- OCR being too slow
- Google speech to text being too slow
- Parsing OCR data
Accomplishments that we're proud of
We worked with tools that we never used before. And everything mostly worked (although somewhat slowly). Overall, the project is intended as either an assistive program for those who need it, or a simple handy tool.
What we learned
We learned how to use many new tools.
What's next for PyVoiceControl
PyVoiceControl needs to be more user friendly and run more efficiently.