Robotic Musical Glove

Inspiration

Learning a new instrument can be very intimidating or demanding. Many traditional instruments need years of practice, expensive equipment, and different styles/methods for different instruments. We wanted to create a system that could remove that barrier for people new to music, to create a fun way for anyone to create their own music instantly, literally in the palm of their hand! We were mainly inspired by our idea that music should be available to everyone, no matter if they are beginners, casual users, or producers looking for a unique, creative way to make beats or melodies.

What it does

The Robotic Musical Glove is a wearable system that takes finger touches and hand motions to produce music. There are capacitive touch sensors on four of the fingers (index, middle, ring, and pinky), and these are mapped to sets of four notes with three different configurations of the hand, which adds up to a total of 12 notes (the number of notes in a full octave). When the user taps one of these four fingers to their thumb, it triggers a note-on event and plays a note on a digital instrument. There is also an onboard IMU motion sensor attached to the inside of the palm of the glove to track movements in wrist motion. If the user rotates their wrist and hand across the forearm axis (roll), the glove will switch note banks and switch from one set of four notes to another (either higher or lower on that current octave). If the user moves their hand upwards or downwards (pitch), the set of 12 notes will change in octave, either going up an octave (hand moving upwards) or moving down an octave (hand moving downwards). This set of actions is done by reading in the acceleration and gyro data that the IMU senses from the user’s movement. This allows us to use sensor fusion by combining both the data sources we are getting from the capacitive sensors and the IMU sensor. Instead of using just one of these sensors at a time, we use both to see exactly which finger is being pressed and how the hand is moving to allow the glove to do multiple complex functions. To filter this data, we use a Madgwick filter to counter the noise we receive from any shaking or drifting of the hand. Madgwick watches the accelerometer and uses gravity as a reference for the downward direction. It then checks to see if your hand is tilted wrong using the gyro estimate, and if it is wrong, the Madgwick nudges this orientation back to what gravity is saying is realistic. This allows us to use this sensor fusion to return accurate values instead of slowly getting "out of tune". To actually play music, the glove connects to software like GarageBand via MIDI, allowing the user to play many instruments, such as piano, drums, violin, bass, and more, using only one hand.

How we built it

We built this project around an ESP32-S3 microcontroller because of its strong processing power as well as USB support. We added the capacitive touch sensors for the four fingers by having four wires each running under one of the four fingers, and then having an exposed end of a wire to detect conductivity. We then connected an IMU from the microcontroller and placed it in the center of the palm in the glove to best track the roll and pitch of the hand. All of the wiring and the circuit were placed between two gloves to make one big glove. For each of the finger tips, we spliced the wires and then placed aluminum foil on them to best detect a spike in conductivity. We then tested several times to create threshold values to accurately detect a finger touch and output a note-on event. For the software part of the glove, we created an algorithm to map the fingers to specific MIDI notes and then use wrist rotation/movement for bank switching or moving octaves. We also tested threshold values for the rotation/movement (roll of the hand) with the angle detection for bank switching and acceleration values for the pitch of the hand to change octaves. The software uses sensor fusion techniques and uses multiple sensor signals at once to allow for these advanced controls/features on the glove.

Challenges we ran into

One of the biggest challenges we faced during the hackathon was building the hardware circuit for our glove. At first, we decided to use perf boards since we thought we needed resistors for the IMU and capacitive sensors. During this phase, we had to solder two perf boards onto the ESP32-S3, and the process became very risky and quick. Many of the pins we were soldering were very close together, and even a small solder bridge could cause a short to happen. Unfortunately, a short occurred, and the ESP32 was fried after we had spent a whole day trying to finish the circuit around it. This was our first huge setback and made us rethink how we wanted to build the system.

After brainstorming some more and troubleshooting, we figured out that the resistors weren’t needed, and a simpler design that didn’t require the use of perf boards would work perfectly. We began on the second day and started by wiring directly from the ESP32 pins to the capacitive sensors and IMUs. This made it much cleaner and easier to place inside the glove as it didn’t require extra wiring. For our second ESP32, we successfully rebuilt the circuit, and it functioned correctly and could output accurate values without shorting.

A second challenge we faced was trying to build the prototype of the glove and figuring out where to put everything. We had to think about how to place the sensors inside the glove and make it functional and comfortable for the user. This required multiple tests as we also had to make sure the sensors could function while inside the glove. We also ran into many issues with sensor noise and random inputs. Small hand movements that were made while we were wearing the glove would give us junk values or sometimes just 0 as the output. We had to fix the algorithm and filter the garbage data that we were getting from both the capacitive sensors and motion data from the IMU. Although we faced these setbacks, we were able to use them as learning experiences and push forward to improve our design, which helped create a more efficient product.

Accomplishments that we're proud of

We successfully used both capacitive touch sensors and an IMU's motion controls to create a functional system with a glove that can play a song using many different instruments on GarageBand.

What we learned

We learned that sensor fusion is a great way to combine different forms of input data and turn it into a working system. In this project, we worked with both touch sensing and accelerometer and gyroscope data to perform different complex functions. We also got to practice and gain experience with embedded systems, data filtering, wearable hardware, circuits, and MIDI communication.