Talk2Control

An online application that controls your screen with voice recognition and filters.

Comment

Inspiration

As students who create and watch endless presentations a semester, we witness a prevalent and frankly awkward epidemic. When showcasing a presentation, most speakers use powerpoints to display information on an array of slides. To flip through each slide, there is often a pause and sometimes confusion as to who or how the presentation is to be moving forward. We decided to create Talk2Control (T2C), an online application that can fix this issue by controlling your screen with voice recognition and filters. The solutions T2C solves is the hiccups of not knowing who/when presentations should be clicked and removes the need for a remote clicker. Productivity is heightened and accessibility issues are addressed. The 4.2 million Canadians with arthritis will no longer need to worry about a bumpy presentation.

What it does

The application is able to detect a pre-recorded keyword so if the user is presenting and says the specific keyword, T2C triggers a command that the user has preset. After visiting the talk2control.tech site, the user is able to insert a block of text that they plan on saying during the presentation. At this point they are also able to record themselves saying a specific keyword that is used within their presentation. The last step is to select their preferred command whether it be to right click, left click, scroll up or down, or press the keyboard. The user controls when and where their commands are placed and have the control to add as many commands as they would like.

How I built it

Initially, we wanted to use ReactJS for the front-end and Flask/Python for the back end. But as time went on we found out that the integration between React and Flask was not sensible and we would not be able to fix those bugs in such a short amount of time. So we altered it and instead wrote pure JavaScript with Flask as a server.

We also used P5.JS to record speech from the user and then get a string from it. Then we sent that text to the server and simulate a keypress.

Challenges I ran into

We attempted to use ReactJS initially but encountered problems. Then, we also tried using NodeJS, but we ultimately decided to stick with just JS because of the problems we encountered. We also had some issues setting up Flask at first but were luckily able to solve this problem fairly quickly.

Accomplishments that I'm proud of

We are proud of our ability to collaborate as a team to solve various problems we encountered. One of us had never gone to a hackathon before, and still contributed a lot. We are also proud of our flexibility as a group, since we changed our entire project from using NodeJS to not using it at all half-way. Finally, we are proud of our final product, which took a long time to program but is something we believe can help us all in the future.

What I learned

We learned a lot about hosting web apps using Flask. It's something we have never experimented with before and took a while to get set up. We also learned about dividing code up as a team and collaborating together. We actively used Github to share our code and occasionally encountered merge conflicts, but we ultimately learned how to properly fix them so we could continue with our project.

What's next for Talk2Control

T2C’s future entails controlling presentations with phrases instead of a single keyword. As presenters upload their presentation speeches, they will be able to highlight certain phrases which T2C will be able to identify with voice recognition and be able to use those verbal cues to execute a command. Additionally, T2C currently detects any voice that speaks into the microphone, but we hope that it would be able to recognize specific voices so only the presenter is able to control commands rather than potential background noise.

Share this project:

Updates