Feel The Music

LeLamp Tutor and Companion
Frontal Lobe High Frequency Haptics
Pre Frontal Lobe Spinal Low Frequency Haptics
Full Experiential Kit
iPhone App for Sync and Haptics

Updates

Inspiration

Music is one of the most social activities people engage in.

People dance together, go to concerts together, sing songs together, listen to the bass in a room, and respond to the beat at the same time.

However, the predominant assumption of music technology is still:

You can hear it.

This means Deaf and hard of hearing people can be denied a moment in which they are not themselves but in the moment with everyone else.

We wanted to change that.

Instead of asking:

How can you make music louder or easier to hear?

We asked:

What if music could be heard throughout the body?

So we created Feel The Music.

A system that converts music to touch, light, movement, rhythm and mutual interaction.

The intention is not to take the place of sound.

The aim is to bring music to a wider audience.

What it does

Play music on Mac.

Switch the speakers to the OFF position all the way.

The music can still be heard in the room.

The music part(s)	Your experience of the music
Bass	Physical haptics
Drums	Rhythmic vibration
Melody	Light and robot movement
Beat	Synchronized motion
Energy	Visual choreography
Other people	Shared interaction

A robot lamp dances to the music.

Your phone starts vibrating, in sync with the beat.

Your action becomes a part of the experience.

Several people can play different parts of the same song, but be in sync with each other.

What's crucial is not the individual feature.

It is the same moment that everyone is in.

Why this matters

Often accessibility is seen as providing the same information in a different form.

We're not sure it can be that ambitious.

Music is not a piece of information.

It is culture.

It is friendship.

It is concerts, parties, dancing, memories and being a part of a group.

The issue with Deaf and hard of hearing people is not just:

"Can I detect that music is playing?"

What could be more important is:

"Can I have this with anyone else?

This modified our thinking on the project.

We're not aiming to create a better vibe.

We're working to create a multi-sensory vocabulary for music.

Instead:

$$ \text{Music} $$

$$ \text{Sound} + \text{Touch} + \text{Light} + \text{Movement} + \text{People} $$

How we built it

If a shared experience is going to take place, then all of the things have to coincide.

When someone feels a kick drum after someone else saw the light flash, it does not work.

So we created the FT Maker Conductor.

The Mac serves as the orchestra's conductor for the whole room.

All devices, including the robot, are synced to the same clock.Phones, haptics, lights, and robot movement all share the same clock.

We also built:

The Local Area Network (LAN) consists of an internal communications system between two or more devices.

synchronized haptic playback
a wearable haptic system
motion tracking
robot choreography
a robot safety layer
physical simulation This is a multi-agent development session with Claude, Codex, Gemini, and GPT.This is an experiment in multi-agent development with Claude, Codex, Gemini, and GPT.

The social aspect is made easy with the engineering.

Challenges we ran into

The final problem was the assumptions of the software were not necessarily accurate when dealing with real hardware.

A robot API recommended the range of 360°.

The actual movement of the physical robot was limited to only moving around by 147° before the alarm would go off.

The robot's center of mass also shifted significantly during fast movements, allowing some choreography to be unstable.

So we decided to create our own implementation and safety layer for the true physical limitations of the robot.

Networking caused another issue.

Public Wi-Fi was not designed for use for low latency device-to-device communications, so we built our own private network.

We also found that AI agents can be certain to repeat their own errors.

We didn't use directly generated code, but had agents review and check each others work before commands were sent to the physical hardware.

We are proud of our service dogs.We are proud of our service dogs!

We wanted the technology to blend into the experience.

That meant keeping the system in sync.

Result	Measured
Overall synchronisation	~6 ms
Bass synchronization	~4 ms
Bass dropouts	0
Robot motions validated	312
Unsafe motions rejected	14

Those numbers are important because once things are synced people don't think about phones, networks, clocks, and robots.

They simply "sing together.

That is what we are interested in.

What we learned

We began the hackathon with the question:

How can one hear music without being able to hear?

At the end we wondered, "is the better question:

What can be done to get everyone in a room to share in music?

Music is not just sound.

It is feeling bass down the floor.

It's observing someone dance.

It is expecting a decline.

It's watching one room respond as one.

The understanding that others in your environment are in the same moment.

This is a place where accessibility is important.

Not only because it provides someone with another signal.

It provides someone with another way to belong.

What's next

Adding more features is not the next step.

It's constructing this with Deaf and hard of hearing persons.

We want to understand:

what aspects of music do people want to feel
where they desire it
what visual information is helpful Which of them don't mean anything?Which of these do not have any meaning? How music is used by people today.
how the use of sign language and movement might be integrated into the interaction.
how a shared musical experience becomes 'social'

First, we attempted to produce music without sound.

The more significant opportunity is to make music multi-sensory, and by that I mean it should just be that way.

It can vibrate with the use of a phone.

It can be displayed with a light source.

It can be moved to by a robot.

You can wear it and feel it.

And then everyone in the room can sing the same song together.