WebSight

Popup

Inspiration

According to the World Health Organization (WHO), over 2.2 billion people across the world are visually impaired.

“The world faces considerable challenges in terms of eye care, including inequalities in the coverage and quality of prevention, treatment and rehabilitation services; a shortage of trained eye care service providers; and poor integration of eye care services into health systems, among others.” (WHO, 2019)

Despite average screen times being at an all time high in the modern day, not all text is effortless to read for everyone. For example, research shows that individuals with low vision due to Age-related Macular Degeneration benefit from in-line text highlighting on webpages. The ideal webpage text is different for everyone, which is why we decided to create a personalized browser extension for this purpose.

What it does

Websight is a personalized, self-learning web-vision assistant that manipulates webpages to the preferences of the user. For individuals with cognitive or visual impairment, surfing the web can be a painstaking task. The days of straining your eyes reading through dense papers are no more!

Features:

Seamless reading aid - Enhances the text you look at on any webpage

Personalized experience - AI learns your preferences and adjusts settings to accommodate for any situation

Self-Optimizing AI - Track eye strain, micro expressions, and user activity to optimize your browsing experience.

Data Privacy - No personal data is stored on servers, everything is saved into local storage

How we built it

Frontend: We made the cross-browser extension using the WXT framework which provides typescript abstractions and utilities for building MV3 extensions. Our eye tracking module gets injected into web pages via a content script and polls data back into the extension service worker.

Backend: Our backend was a FastAPI python server which contained API routes for interacting with gemini and our Pytorch model.

Challenges we ran into

many of the eye tracking tools we tried were not accurate enough for our project and we tested a couple before settling on WebGazer.js.

Accomplishments that we're proud of

Combining face tracking and eye tracking to accurately track where the user is looking at on a web page and their facial expressions
Using AI to analyze user preferences