Collective Cadence

• 

  virtual_keyboard

• 

  output

• 

  piano

• 

  drum

• 

  block diagram

• 

  output from our auto transcriber

Inspiration

We were inspired by our own experiences jamming out to music on our own, but being unable to easily transcribe it in the form of sheet music and share with other people.

What It Does

We created a remote pipeline for taking analog signals of instrumental music, writing matching sheet music and MIDI files, and streaming this music remotely.

We built two electronic instruments (an electronic keyboard a mobile set of drumset sensors) for prototyping the uses of this pipeline. The keyboard captures both rhythm and pitch information, and remotely sends this info as MIDI files to a WebSocket server. The drumset data is processed on Matlab and LilyPad to be converted into MIDI and sheet music.

How We Built It

We had three main parts: collecting data of drum beats and building a keyboard through ESP32. using Matlab and LilyPond to process this with machine learning and traditional techniques to get the sheet music transcription. setting up Websocket to allow this entire pipeline to run remotely.

Challenges We Faced

We struggled to use ESP32, especially with Arduino code. We also faced a lot of challenge from noise and came up with many ways to reduce it through hardware or eliminate it through postprocessing.

Accomplishments that we're proud of

We're proud of how this project increases both accessibility and community. We aim to reduce the skill barrier in transcribing and sharing improvised music and jam sessions. This product is unique in the field of music transcription software because it features customizable rhythm correction and a livestream feature to better connect with other musicians.

What We Learned

We researched a lot about vibration sensors, machine learning, MIDI files, and WebSocket. We learned to identify and solve the problems that can be caused by equipment setup, such as sample rates being limited by button debouncing and the resonance of the table being picked up by our sensors.

What's next for us

We hope that we can fully integrate both musical instruments into the wireless remote protocol; which was a barrier we faced from the parts availability at the hackathon.

Built With

• arduino
• esp32
• lilypond
• machine-learning
• matlab
• midi.js
• websockets