EyeReplace

This shows the tech stack we used, with arrows representing websocket-io pipelines.

Inspiration

EyeReplace is committed to advancing equitable education by ensuring that everyone, regardless of their reading ability or language proficiency, can access and comprehend online educational content. Many individuals face barriers to learning due to reading comprehension difficulties, whether stemming from learning disabilities, language barriers, or the challenges of developing literacy skills at a young age. For those with dyslexia, which affects over 700 million people worldwide, and for non-native speakers navigating unfamiliar vocabulary, these obstacles can significantly hinder learning opportunities. EyeReplace offers an innovative solution designed to enhance readability, empowering all learners by fostering a more inclusive, accessible web. By breaking down these barriers, EyeReplace ensures that everyone has equal access to educational resources and the ability to engage with knowledge on their own terms.

What it does

EyeReplace is a chrome extension that uses the Beam Eyewear SDK in tandem with a python to track a user’s eye movement, detecting when they pause on a word and replacing the word with a simpler one, using JavaScript on the chrome extension. To help accomplish this, EyeReplace modifies web pages to larger font sizes and a distraction free layout for easier reading. Then, when it detects the user as pausing on a word, it replaces the troublesome word with a simpler, more meaningful alternative. It replaces the word with either a simpler synonym, a translation from the user’s native language if they are English as a Second Language (ESL) classified, or a dyslexia friendly synonym. Meanwhile, the extension also collects data on words that frequently cause difficulty, providing insights, questions, and flashcards through an interactive website with user authentication and well-designed UI.

For classroom use, teachers can monitor students’ progress in learning new words through the website, where they can adopt a teacher role and view the weekly statistics of all of their students with the student role. These students can consistently improve their reading comprehension abilities through the provided resources, allowing them to grow their skill AND comprehend the text they’re reading simultaneously. EyeReplace not only makes reading easier but also provides crucial support for those with dyslexia and ESL, fostering a learning environment that honors and accommodates their struggles to support them towards a more successful future.

How we built it

EyeReplace integrates a Python backend server with a JavaScript frontend implemented as a Chrome extension. The backend is powered by Python and uses the Beam Eyeware SDK to track real-time eye movements, with a calculation algorithm that pinpoints when and where a user encounters difficulty while reading. When a user’s gaze lingers on a word, the system interprets this as a moment of confusion. The backend then sends the gaze coordinates to determine which word the user is fixated on, processes this word through a large language model (LLM) or external API, and suggests a synonym or simpler alternative to enhance comprehension. The word and its synonym are stored in a database for personalized tracking.

On the frontend, the Chrome extension uses JavaScript to monitor browser interactions, captures the user’s gaze data, and sends it to the backend. When the backend returns the appropriate synonym, the extension dynamically replaces the word in the text, ensuring a smooth reading experience. Users can customize their reading preferences, and these adjustments are saved in the backend, making the reading experience tailored to individual needs. One of our major limitations as of now is our lack of ssl-certification; we are limited to HTTP websites for applications of our software. However, the fix is easy as we have tested HTTPS in a localhost and seen that it works fantastically. We simply need to acquire entrepreneurship; which isn’t that difficult.

We used Google Authentication to securely give users access to their own statistics and record logins on MongoDB Atlas. We have authentication on our web app and authentication on our chrome extension that point to the same api endpoint, allowing for data to be stored for users in our database. We have teacher roles to give instructors access to student performance data, allowing insights into which words a user struggles with, and giving students the opportunity to improve through our learning features such as quizzes and flashcards.

Challenges we ran into

It was very hard for us to find a good eye-tracking software that had a high accuracy and also produced reliable inputs for our backend processing. Finding a balance in beam SDK eye tracking software, we hoped to achieve desirable results. Due to the complexity of the software, it was hard for us to pinpoint the user’s attention on a screen to a singular point to identify the word. We were able to overcome the issue by using better calibration and algorithms to actively track user attention to a point. Moreover, integrating web sockets to facilitate communication between the frontend and backend of our product proved to be a hurdle. Since our application sends and receives multiple data calls, merging the various components of the project was complex. We used a JavaScript-based front end and a Python-based back end, which required robust middleware that we were initially unfamiliar with. However, through collaboration, peer programming, and caffeine, we managed to build a reliable backend web socket server that allowed our front and back end to effectively communicate for our purposes.

Accomplishments that we're proud of

One of our most significant accomplishments is integrating multiple data pipelines to create a practical solution that aims to revolutionize online surfing. By leveraging APIs, we didn't just replace words but also improved the context to enhance the overall experience for target users, increasing the accuracy of word replacement. Our solution encompasses a broad use case, involving various components such as APIs, eye-tracking software, websocket servers, a Chrome extension, and a website. This holistic approach offers more than a short-term fix for word replacement; it provides a long-term solution that helps students and ESL learners improve their vocabulary, making it a valuable educational tool.

What we learned

EyeReplace wasn’t just a project for us, it was an opportunity for personal growth. We recognized that building something this complex without prior knowledge would not be easy; bedridden with obstacles along the path. But we decided that nothing is too far-fetched, no idea is too crazy, and getting no sleep is okay for a weekend. We learned that if we genuinely keep trying at something, through iteration, redirection, and collaboration, we can accomplish anything we set our minds to. We experienced firsthand the value of resilience, perseverance, and determination in the face of adversity. And through it all, we learned career-defining technologies and skills that we will carry throughout careers at Georgia Tech. Building EyeReplace was invaluable to every member on this team, and we are grateful to have been given the space and opportunity to do what we did.

What's next for EyeReplace

Looking ahead, we are excited about the future of EyeReplace and the innovative developments on our horizon. One of our primary goals is to harness the power of machine learning to analyze user reading patterns. By doing so, we aim to identify potential words that users may struggle with, further enhancing their reading experience and making our tool even more intuitive and responsive to individual needs. Machine learning would allow our words to be much better and cater more accurately to the user’s specific needs.

In addition, we plan to extend EyeReplace’s features to allow users to upload their documents directly to our platform. This functionality will enable us to modify these documents for clarity and ease of understanding, making it particularly beneficial for complex materials like post-operation documents related to patient safety, financial documents, and intricate school assignments. Reading comprehension is involved in every industry, and people across subjects deserve to be able to read. Moreover, we are exploring the integration of augmented reality to transform how users interact with their environment. By overlaying helpful information onto the physical world, we can create a more immersive and accessible experience, helping users navigate and interpret various elements in their surroundings with ease.

With these advancements, we envision EyeReplace not only as a reading aid but as a comprehensive tool that empowers users to thrive in both their academic and everyday lives. We are committed to continuously innovating and improving the user experience, making reading and comprehension easier and more engaging for everyone.