Flipping through music sheets is incredibly frustrating. A person has to take their hand off an instrument to turn the page and often loses their spot. Musicians desperately avoid ever having to turn a page.

One day we thought "how can we solve this problem? How can we avoid using our hands to turn a page?" and the idea for Muvïc was born.

What it does

Basically, You play, Muvïc moves. While playing an instrument Muvïc uses facial cues to know when to move through the music sheets. Musicians no longer need to use their hands to turn pages.

Muvïc lets users select sheet music from their account collection. Now, what if a user wants more sheet music in their collection? Muvïc allows users to upload their own XML music sheets or lets them explore a variety of musical sheets available for download through the online shop.

Once a user has selected a piece to play, Muvïc will ask for access to the device's camera. Once Muvïc recognizes a user's face, they will be able to use facial cues through elaborate facial recognition algorithms. It simply takes an eyebrow raise to move forward.

Try it now: Muvïc

How we built it

The essentials

All projects need good design and support.

  • The web and mobile application development was made possible with Firebase.
  • Hosting for the web-app is made possible by Amazon Web Services (AWS).
  • Web development and design is done with javascript, html, and css.
  • Python and Node.js for the back-end server.

The Muvïc Magic

The crux of the project is manipulating a music sheet without your hands.

One way this is accomplished is with a facial recognition API, Affectiva. We use Affectiva to gather data points of a user's face from a device's webcam. This is done in javascript using Affectiva's javascript SDK. Based on this information we manipulate the page (stored as an XML file) to show the desired lines of music.

Alternatively, measure tracking moves at tempo with the music. This is done by scanning the XML page using octave and interpreting the music on the page to locate the different measures.

AJAX is used to manipulate the page (scrolling down and/or highlighting measures) for both cases.

Challenges we ran into

The things we found most difficult and time consuming include:

  • Interfacing Afffectiva with our webpage and then taking those Affectiva responses and using them to manipulate the sheet music in real time.
  • Parsing XML using Octave to identify notes, variations and tempo.
  • Node.js because we originally tried to parse music sheets in PDF instead of XML

Accomplishments that we're proud of

At the end of the day, what matters the most is we built something cool for something we're all passionate about. We worked together at this hackathon because of our shared passion for sound, music, and digital-audio. We saw a problem in our community and spent the weekend developing a viable solution we would all use.

What we learned

Krista learned how to:

  • make post and get requests between the Node.js server and JavaScript code
  • manipulate JSON responses from an API
  • effectively use git from the command line when collaborating with other people
  • the importance of version control.
  • set up AWS instances

Anthony learned how to:

  • use Google material design
  • how facial recognition can be used in web-apps
  • build and design using a response approach first
  • authenticate and mount on a aws-ec2 instance

Fengchi learned how to:

  • write split timed information pulling
  • use Firebase
  • manage overflowing overflowing variabes
  • understanding code with poor documentations

Pierce learned how to:

  • use Firebase
  • use Google material design
  • page recognition

What's next for Muvïc

In less than 36 hours Muvïc grew from just a sheet music turner to a full fledged web-based sheet music manager. We would love to add and/or expand on functionalities that will make Muvïc the best all-in-one web-app for musicians.

In this spirit, we hope to:

  • Extend the acceptable format of music uploads to include PDF -- This will increase accessibility of sheet music
  • Scroll backwards using facial cues -- This will allow users to get back to the right place encase they scroll too far using the facial recognition - without ever leaving facial recognition mode.
  • *Turn it into a cloud based SASS * -- Because it will allow users to have their own logins and portals.
  • Download sheet music from inside the application -- We want to link our web-app to online vendors and sheet music distributors that have sheet music available for download in the acceptable formats. -- This would let users search for new music they want to learn directly from within the app
Share this project: