GazeScore

As a musician—PAGE TURNING SUCKS!You miss your turn, or flip too many—it’s a mess.That’s where we come in with GazeScore—keeping your next line in sight, so you’re always ready to play!

Comment

Inspiration

One of our team members (Manit) has been a classical pianist for the past 10 years, and page turning has always been a frustrating challenge. This year, while recording a 40-minute piano performance, he lost two fantastic takes—all because of messed up page turns. It was a small mistake with big consequences.

When brainstorming for the hackathon, we asked: What would the ideal solution look like? The answer was simple—page turning that feels seamless and natural, always showing the next line exactly when you need it. That’s what inspired us to create GazeScore.

What it does

The website has two main sections: Setup and Perform.

Setup: The user uploads a PDF of their sheet music and completes a quick eye-tracking calibration to ensure accurate gaze detection. Perform: The sheet music is displayed split into two halves, allowing for smooth transitions.

Here's how it works in action:

  1. The musician starts reading the first half of the page with no interruptions.
  2. As they shift focus to the second half, the system detects when they've been reading it for three seconds (our threshold for confirming intent).
  3. Once confirmed, the first half of the next page is preloaded, ensuring that by the time the musician finishes the second half, they seamlessly transition without missing a beat.
  4. This chain continues for each half, ensuring that transitions remain smooth and uninterrupted throughout the performance.

This method ensures a frictionless page-turning experience, eliminating delays and allowing musicians to focus entirely on their performance.

How we built it

We developed GazeScore as a React + Vite web application, integrating multiple libraries to handle PDF processing, eye-tracking calibration, and seamless page-turning logic.

  1. Development Process Before writing any code, we spent an hour mapping out our entire project on a whiteboard, brainstorming ideas, defining core functionality, and even considering future features. This structured approach helped us prioritize tasks efficiently. We then delegated independent tasks to maximize productivity, allowing us to work in parallel while ensuring all components seamlessly integrated.
  2. PDF Upload & Processing We used pdf.js and pdf-lib to handle PDF rendering, formatting, and slicing, allowing us to split sheet music into halves for a smooth reading experience.
  3. Eye-Tracking Calibration We integrated WebGazer.js for real-time eye tracking, but with a customized calibration setup that ensures the system accurately detects gaze positions across all possible points on the screen. This fine-tuned calibration improves accuracy and reliability during performance.
  4. Smart Page-Turning Logic In Perform mode, we implemented a custom logic to track the musician’s gaze and trigger page turns dynamically:
  5. The screen is divided into top and bottom halves, and timers track how long a musician focuses on each section.
  6. When the musician looks at the bottom half for a set duration (3 seconds), the system preloads the next top half. Vice versa, when the musician looks at the top half long enough, the system preloads the next bottom half.
  7. A boolean variable manages which half of the next page should be turned next, ensuring that the flow remains smooth without unnecessary page changes.

This logic continuously updates throughout the performance, eliminating manual page turns and keeping the music flowing.

Challenges we ran into

  1. We had a lot of issue with rendering the PDF, and we had to debug our worker.
  2. We had to develop our own logic for page flipping, as this wasn't something that existed before.
  3. Our PDF splitting required us to convert from C# to Javascript.
  4. We had trouble integrating the CSS files to not override each other.
  5. Defining the calibrating system/integrating the WebGazer API to accurately track eye movement.

Accomplishments that we're proud of

  1. Integrating the eye-tracking was very difficult, but came out really well.
  2. Our UI came out well, and has a consistent style across the website.
  3. We developed our own logic for page flipping and create code for it all by our selves.
  4. In the past we were never able to apportion tasks in a project like this efficiently, and we learnt how to do so.
  5. Creating a true MVP (minimum viable product) that solves an issue that many people face.

What we learned

  1. We learnt how to create React apps using Vite.
  2. We learnt how to develop eye-tracking software using APIs.
  3. We learnt a lot about UI design and developing an eye-catching website while being efficient.
  4. We learnt about project management, splitting up a big task into smaller bites all of us could tackle at the same time, and combining our sections into a final product.

What's next for GazeScore

We're excited about the future of GazeScore and have several features in the pipeline to enhance its reliability, usability, and scalability:

  1. Audio Recognition-Based Page Turning – Integrating sound detection to further improve the accuracy of page turns, ensuring seamless transitions even in cases where gaze tracking might be less reliable.
  2. Refined Page-Turning Algorithms – Upgrading our logic from a moving window approach to a sliding window, improving responsiveness and reducing false triggers.
  3. User Accounts & Cloud Syncing – Implementing a Firebase backend to support authentication, user profiles, and cloud-based settings, allowing musicians to save their preferences and sheet music.
  4. URL-to-PDF Conversion – Adding a feature that lets users convert web-based sheet music into PDFs directly within the platform, making GazeScore even more versatile.

Additionally, we see potential for premium features, such as cloud storage and advanced customization, opening opportunities for monetization and long-term sustainability.

Built With

Share this project:

Updates