FlipThatSheet

Inspiration 🎹

Imagine you're a pianist. You've practiced for countless hours to prepare your magnum opus for concert day. It's your moment, your time to shine on stage. Things seem fine at first as you start your performance… But Oh Sheet™! There's a page turn coming up! As you fumble to turn the page on your tablet, you lose your flow, and ruin your big moment.

As musicians ourselves, we ran into situation too many times. So, we developed FlipThatSheet! A music importer web application... that can flip pages completely hands free!

What it does 🎼

FlipThatSheet allows for users to upload their music pdfs to their accounts for personal viewing. While in viewing mode, the user can tilt their head left or right to turn the page left or right, as opposed to using their hands to turn the page manually. In addition, a login system lets users keep their music collection private.

How we built it ⚙️

Our project features a frontend built with Next.js and React, styled using Material-UI. The backend includes a Python/Flask component for integrating the AI model with the frontend.

We also implemented MongoDB and Auth0 alongside Express.js to store imported pdf references and store login/signup.

The head tilting detection in our project was implemented using TensorFlow, OpenCV, and Mediapipe. We utilized the Mediapipe library for pose and face detections through OpenCV. To train the ML head tilting model, we recorded 400+ videos, using the pose and face landmarks extracted from Mediapipe as training data. The model was trained using TensorFlow. We employed a Redis server to handle the predictions made by the ML model, which were sent to the frontend via Server-Sent Events (SSE). The frontend controlled page turning with the react-pdf module and Next.js.

Challenges we ran into 💪

Lot’s of the tech stack that we decided to use ended up running on different backend servers such as flask for Python, Express.js for MongoDB and Auth0 and Redis for flask_sse. Integrating all of them into one frontend framework proved to be way harder than we initially thought.

In addition, we initially started our project with react-native instead of Next.js, however realized later on that it wouldn’t support the packages we needed to use. So, halfway into development, we were forced to scrap the react-native frontend and switch to Next.js

Lastly, training our ML model was a tedious and time consuming process. Any bad data in our dataset would completely ruin our training, and force us to completely retrain our model with new data. Our model turned out great, but our necks may never recover.

And also pdfs are a pain to get working on Next.js for some reason.

Accomplishments that we're proud of 🏆

During our hacking period, we were able to: Train a ML model with 97% accuracy Set up a MongoDB database that stores pdf references for different users Set up a login/signup system using Auth0 Setup and use a Linux virtual environment to run a Redis server Almost blow up a laptop by running three servers and setting up the aforementioned virtual environment

What we learned 📚

There were many firsts for us three during this event, for example, MongoDB, Redis, MUI, and Flask, which was a stressful but valuable learning experience. Due to time limitations, we were forced to get scrappy with some of our backend, forcing us to learn how to harness the jankiness of Next.js, Flask, and Retis.

What's next for FlipThatSheet 🚀

For the future of FlipThatSheet, we plan to add more functionality to help musicians stay productive, such as writing directly on the music or a metronome. During development, we were planning to add a recording feature to record and store audio from practice sessions/performances to be reviewed later by the user, but we were ultimately forced to scrap this due to time constraints. There are also more opportunities to use ML with the app, such as a computer vision posture corrector model or ear training audio analysis model.