Inspiration
We really wanted to use Computer Vision and AI to do something that might be beneficial to others. I wanted this app to be a reminder to be more self-conscious about what you are doing, but we made it so it relays data and lets your friends and family know as well. Imagine crunch time at 10 PM at night for a project due at 11:59 PM. Turn Stay In The Know on to keep you focused on the task at hand.
What it does
This project uses machine learning and AI to detect humans and the objects they are holding. We are able to determine what the person is doing and relay a specific command with text to speech to notify the action. It will also send a text message to anyone you want to notify them of what you are doing, anytime anywhere. For example, when working on that project mentioned earlier any phone time should be off limits. Our program can detect any usage of phones and warn the users to put it back down!
How we built it
We used OpenCV for our computer vision and TensorFlow for our machine-learning model. We integrated Google's text-to-speech for it to narrate the actions that are being done, and we used Twilio's API to send text messages to notify others about the user's actions.
Challenges we ran into
The main challenge we had was getting the text-to-speech to work. The text-to-speech either lagged the frame of the computer vision or kept on repeating itself. It is very hard to solve both issues so we tried to compromise as little as we can to insure the text-to-speech is working and it is not too laggy for the computer vision. There was also a lot of permission errors when storing and reading audio files used for TTS.
Accomplishments that we're proud of
We were really proud of what we were able to accomplish with very limited knowledge on computer vision and machine learning. It was our first times using all these new APIs and libraries, but we were able to make it work and combine everything together to get one coherent application.
What we learned
We learned a lot about computer vision and machine learning. It was also our first time using Twilio's API to send text messages and our first time using python to do all these projects. It was a steep learning experience to learn new libraries and read through documentation, but it was a rewarding experience.
What's next for Stay In The Know
It would be great if we were able to perfect our computer vision application to be seamless and run on the GPU. The actions could be improved for more precise detection and there can be a increase of available messaging formats, such as email and whatsapp. In the future it should even be able to detect anything in the world to cater any events requiring people to Stay In The Know!
Log in or sign up for Devpost to join the conversation.