Frankentrumpet

Inspiration

Two of our members play trumpet. We are also inspired to help people with disabilities, the world's largest minority. We combined the two ideas to create a proof-of-concept that manifested as Frankentrumpet.

What it does

The Frankentrumpet has an accompanying website, which takes an image input as sheet music. The image is converted to a MIDI file, which is then packaged and placed in a link in the backend. The MCU reads this using HTTP requests and plays the correct notes of the song in the correct time.

How we built it

Software / High Level

We created a website for our project using React and Express.js and it was deployed with Heroku. The website allows users to upload music sheet image files and sends it to a server which processes data and formats into a JSON-like structure to store the musical notes and timings. It exposes a REST endpoint for the microcontroller to retrieve the formatted music data.

Hardware / Low Level The electrical subsystem of this project consists of an ESP32 Vroom, a power source, and 3 servo motors. These motors are rigged to the trumpet using the materials we had available and combined with makeshift levers to form mechanical actuators that press the keys on the trumpet. The ESP is driven by a script written in Arduino language that connects to the internet and sends an HTTP GET request to the backend of our proxy using a decided URL. The script unpacks a JSON file from the GET request and commands the motors to play different notes using valve numbers and time delay from the JSON file.

Challenges we ran into

Software / High Level

Getting the Optical Music Recognition ML model to work. We were not successful in integrating OMR into our hack. The open-source ML model we were using did not have up-to-date documentation. It would have allowed the server to recognize notes from a sheet music image.

Hardware / Low Level During the building and validation of the basic system components, we had a lot of hardware trouble. Our initial design had DC motors powered by L298N H-Bridge motor drivers, powered by an 8v DC source. We spent hours trying to get these DC motors to function with the motor drivers with no luck. After relentless testing, it was discovered that both of our motor drivers were dysfunctional, so we had to make a hard decision and pivot to using servo motors instead. We also ran into some brief challenges with the Wi-Fi connection from our MCU, which were resolved efficiently.

The mechanical aspect of our project was also challenging. We came up with several different prototypes to hold the servos in place so they could push on the trumpet valves. We managed to improvise a solution for the trumpet finger mechanism using spare parts. Our current solution has some reliability issues, but nothing that can’t be fixed with more time and resources.

Accomplishments that we're proud of

We interfaced from an ML model using a deployed website frontend to communicate with a server backend that could handle HTTP requests from a microcontroller. This controller was able to process data from that request and accurately control motors that created meaningful mechanical output. That is an incredible amount of subsystem interfacing and more actuation than you might be used to seeing at a hackathon. This system, while not working as perfectly as we might hope, is still clear of our expectations of what we thought we might accomplish in 24 hours. The system connects from end to end, and a working product makes a happy team.

What we learned

Along the journey of this hackathon, we learned many things. Even though this was clear and established from the start, it could not be emphasized enough how important it is to quickly validate the base functionality of the system. As we worked on it, we should have made hard decisions about dysfunctional pieces faster to save precious time that could have been spent cleaning up what did work.

What's next for Frankentrumpet

This system is a true prototype. A manual ORM could be developed to read music from an image to fit it with the system, the website could be fleshed out to be more graphically appealing, and most importantly, the mechanical system should be completely changed out to utilize custom parts.