Eye tracker;thinking assistance

While the tracker works across your entire screen the extension assistant using Sonar API responds intelligently to what you're reading within websites acting as a realtime thinking companion.

Comment

Inspiration

We wanted to explore what happens when your computer pays attention to what you look at—no clicks, no commands—just intent through gaze.

What it does

A real-time eye tracker detects where you're looking on screen. A browser extension (powered by Perplexity's Sonar API) reads that context and offers smart, timely help on websites you're viewing.

How we built it

Python with bunch of frameworks for gaze tracking and calibration

JavaScript extension that connects to the Sonar API

Database links the tracker and extension in real-time

Challenges we ran into

-Accurate eye mapping under different lighting -Syncing gaze with dynamic web content -Balancing responsiveness with minimal intrusion

Accomplishments that we're proud of

Functional gaze-to-assistant prototype Smooth real-time communication between native and web components This is a huge project accomplishments for us especially whose that just introduced to this field

What we learned

Attention is a powerful signal, but it’s noisy and fragile.

Making assistants less intrusive but more useful requires restraint, not just intelligence. Bridging desktop-native systems with browser-based tools opened up a lot of creative flexibility.

What's next for Eye tracker;thinking assistance

-Make the assistant smarter: context awareness across tabs, documents, and video. -Expand tracking to full-screen apps (beyond the browser). -Train a small local model for privacy-respecting summarization instead of calling external APIs. -Explore accessibility applications—where gaze might be the only usable input.

Built With

Share this project:

Updates