We all have the dream to be someone or achieve something. To this end, we earnestly plan our time spending and closely track our progress. However, it's very frustrating when we failed to hit our goal, again and again. Due to the problem in this implementation stage, we tend to believe we are doomed to be mediocre. However, with all the social media and possible distractions on the internet, it's really hard to stick to what you are supposed to finish. Understanding and predicting your own behavior pattern is the key to tackle many problems such as anxiety, time management and evaluate your productivity. We want to help students to understand the trigger of these distractions and the pattern of their behaviors by automatically tracking and analyzing their computer use.
What it does
In this hackthon, we only try to make this tool records what you are looking at on your computer. From this, it will take all the texts, images, and other data to see what you have been focusing on. From there, you can see where your time has really been spent.
How we built it
In our current implementation, we have mostly used python. As for how it is implemented, we recorded our screen for x minutes. From this, we break the video down into images, where each image is a second in the recording. Then, we run these images into an OCR reader, which (tries) to convert all the text from the image into strings. Finally we take the string data and put it into a matrix. This matrix shows which words are shown at which second, and how many times.
Challenges we ran into
• OCR not being accurate, even after amplifying the images, many characters are still not recognized correctly. • We didn't get enough time to finish t-SNE to group words semantically, therefore the representation of the content is very computationally expensive and not clean for user to visualize the results. • We didn't prepare any labeled data to train the object detection and image captioning to extract information from images and videos. • Setting up the website to demonstrate the functionalities takes longer than we planned. • Security issues of this application isn't considered so far.
Accomplishments that we're proud of
We believe this a great idea although it doesn't take very fancy techniques. First, we are the targeting users and we understand what we want from this application with details. Second, all the techniques it takes to solve these problems are already there, we just need more time to finish them.
What we learned
Doing something not fancy but practical is also very fun.
What's next for MyTime
Basically solve the challenges we have met and try to infer more specific conclusions from the data we extracted. More specifically, we will:
- increase words extraction accuracy
- train CNN/NLP for image annotation and semantic analysis
- combine the multi-modal data of words, image and caption to further improve the content extraction accuracy
- design user interface to customize the content classification first
There are several potential directions:
- mood detection and chat bot
- personal assistant for data retrieving