eyenote

Inspiration

As students we know the challenge of sitting overwhelmed in a lecture as more information than we have time to process and write down is being presented. We wanted to build a tool that allows students to focus on the lecture and not worry about taking notes.

What it does

eyenote tracks eye movements and captures field of view to auto generate notes in a google docs file based on the lesson you’re watching. It also tracks pupil dilation to determine interest / difficulty/ focus in specific sentences/ topics and add summaries and definitions to those topics. Finally, it provides recommendations about your learning in a pop-up based on the analyzed eye tracking.

How we built it

We use custom LLM-driven OCR tools combined with the Adhawk's MindLink Glasses to pinpoint exactly where a user is looking on a screen and employ direct pupil measurements to estimate drive and positive emotion. We then use google docs APIs to create documents with text from the screen and additional text based on estimated drive. Ultimately, we leverage dynamic prompt templates to harness both user vision and text information with Langchain as part of a Python-Flask backend, providing specific and neurally-tailored recommendations to the end user on a ReactJs frontend - catalyzing performance.

Challenges we ran into

Our idea relies on eye tracking along with imaging of the field of view so that the information being tracked can be used. However, part way into creating the project we discovered that the adHawk glasses do not come with a camera. Instead, we had to create a baseline starting point for the user and track their relative up, left, right, and down positions and then project those positions onto the device being watched to know what quadrant the user is looking at

Accomplishments that we're proud of

Firstly, we are proud of our teamwork and organization. Despite the limited time of a hackathon we made sure to set deadlines, discuss approaches with each other and provide valuable feedback.

We are also proud that we did not get discouraged and found a workaround when halfway through the hackathon we discovered that our plan to build our project was not feasible since the AdHawk glasses did not work as anticipated.

What we learned

We learned that it's very important to narrow down on a specific topic / issue that we want to build or fix, rather than picking a domain and trying to narrow it down from there. We found that when picking a domain, the options are endless and it makes it very hard to pick one and narrow down enough to pick a feasible project.

What's next for eyenote

With more research and time, eyenote will further understand the connection between the eye and the mind state of a person; this information will be used to further customize the augmented notes created. For example, there will be more distinction for when a summary/ definition should be added versus related novel information. As well, the ML model can provide more personalized recommendations with further training. On an even grander scale, this project was the refinement of our collective interest in the applications of behavioural science and psychology with AI and machine learning.