As students, acclimating to the new environment that is virtual school has not been easy. With the dozens of reasons to be stressed and the nature of virtual lectures, it is very easy to find ourselves drifting our attention away from the lessons and classes. With the absence of direct interaction and the classroom environment, the probability of distraction skyrockets. While your professor or friends may no longer be next to you to snap you back into focus, we have taken it upon ourselves to create an app that does that and more. Everybody could use someone or something to keep them accountable and that's what Refocus is here to do.

What it does

Refocus currently has two primary functional tasks.

Using a Tensorflow javascript face-recognition web API, the user can turn a camera to monitor their movements and attention while watching a video, studying, or watching a lecture live. The API is applied so that if the user looks away or is no longer in the camera's view, they will be prompted with a visual and aural cue to return their focus back to the screen. In addition, a voice-recognition web API has been implemented so that if certain keywords are said by the lecturer (such as test, quiz, important, assignment, listen carefully, focus, etc.), the user will be pinged and made aware that it is especially important that they pay attention at that moment. There is also an added feature that a summary of emotions experienced through the session is displayed.

Using the same voice-recognition API, Refocus also has a study session feature that serves as a virtual assistant to the user. A user can say anything out loud and Refocus will take note of the time and the content of what was spoken to be reviewed at the end of the session. This enables for less distraction and for making note of confusing words, more difficult ideas to review later, or distractions that came to mind which can be said to the program and put away for later. Intermittently, the program will also remind the user to stretch or drink water. They can also have quick access (via voice command) to certain tools useful while studying such as Spotify, background ambient noise, and productivity suites such as Google Drive. A feature that could be further implemented is well-being checks to prompt the user for how focused and energized they are.

Additional features include a note-writing section and revision time-table/schedule.

How we built it

The frontend of the web app is programmed with HTML, CSS, and some javascript with some initial prototyping and mock-ups in Figma for reference.

The main facial tracking components were done by the Tensorflow.js face-API. Retrieving and formatting data found through the facial-detection was done with javascript. The voice-recognition for the lecture aid as well as for the studying assistant was done by implementing the speech-recognition WebKit so that users can make vocal commands and the webApp can identify special keywords.

Challenges we ran into

Implementing responsive web design became cumbersome at several points, especially when dealing with the web-cam and facial landmarks. We spent a decent amount of time trying to rewrite and redesign the code to make the final product more responsive. In the end, some parts had to be for going to focus on key functionality and larger display components. Another challenge often came with implementing functions and data into our front-end. This was only possible through referencing several sources, documentation, and experimenting with the console to determine the appropriate way to retrieve and manipulate data.

Accomplishments that we're proud of

For many of us, this was our first hackathon and the first use of javascript. This made it a very challenging yet rewarding process to learn from. Even just the act of getting a button in the right place definitely sparked joy within the group. We weren't able to implement all the features we had intended to, but we were fully aware of this and are proud of how much we were able to do in the given time. With what has been implemented, we believe the utility of a web app like this is quite clear and we strongly see how this idea could very useful to any student dealing with the struggles of remote-learning.

What we learned

We believe Refocus definitely has been paramount in developing our understanding of the fundamentals of javascript. Also having implemented an ML API was a great way to get exposed to Tensorflow and its applications. We were able to see that is a very powerful tool that can be used in a multitude of ways once a strong understanding of its libraries and attributes is developed. Getting more familiar with the front-end to create clean and responsive designs was also a skill we definitely developed through this project.

What's next for Refocus

While facial recognition is considerably accurate, we could improve it further by having a more precise eye detection (so that the user's line of sight can be used to identify attentiveness). However, using the more accurate model for face recognition is still beneficial, particularly for the emotion-identification. Further reinforcement could be used to ensure user attention better (i.e. implementing a strike system resulting in a larger consequence such as social media posts). Improving the keyword identifier could also be done by allowing users to add in their own keywords that may correspond to the specific lecture being viewed.

The study session and well-being checks could be used to identify the work habits of students, and thus determine ideal increments of time to work in (i.e. create an optimized and individualized Pomodoro system for students). The voice assistant can also be augmented by incorporating Google Assistant to return more information.

Implementation of the note-taking and revision schedule software (with the application of spaced repetition) would also enable Refocus to play a much larger role in students' productivity and studying.

Last but not least, implementing more server-side functionality would allow us to store data on an account basis over long periods of time to determine better analytics for each user.

Built With

Share this project: