Between Us

Inspiration

Between Us was inspired by the idea of making invisible human experiences—such as brain activity, emotion, and intention—visible and shared. We wanted to explore how music can act as a bridge between people, synchronizing not only sound and movement, but also mental states, within immersive virtual environments.

The project draws from interests in music, neuroscience, and interactive art, aiming to transform abstract data into expressive audiovisual experiences.

What it does

Between Us is a real-time interactive system that combines music visualization, hand-tracking instrumentation, and EEG brain-computer interfaces. Audio, body movement, and brainwave data are transformed into dynamic particle systems, colors, and shapes in Unity.

Users can create music through hand gestures, visualize WAV audio files, and see their brain activity—such as concentration and relaxation—represented visually. In multiplayer scenarios, the system also shows how brain activity can synchronize between participants.

How we built it

The project was built primarily in Unity, integrating multiple input and processing layers:

WAV audio files are analyzed in real time to extract frequency and amplitude data.
Hand tracking (Meta Quest 3 / Meta All In One SDK) is used to generate procedural sounds based on gestures and movement.
EEG data from MindRove headsets is processed using FFT to extract brainwave bands (alpha, beta, theta, gamma, delta).
Unity Particle Systems and VFX Graph are used to visualize music and brain activity through color, size, and motion.
Earlier iterations used Sonic Pi, MIDI, and OSC, but the architecture evolved toward a fully Unity-based solution for stability and performance.

Challenges we ran into

One of the main challenges was dealing with hardware limitations, especially early attempts using Arduino-based MIDI controllers, which were not powerful enough for real-time music generation and visualization.

Additional challenges included:

Processing and cleaning noisy EEG data in real time
Synchronizing multiple real-time data sources (audio, EEG, hand tracking)
Balancing visual complexity with performance in Unity
Iterating on system architecture as technical constraints became clear

Accomplishments that we're proud of

Successfully integrating EEG brainwave data into a real-time interactive visual system
Creating expressive, gesture-based musical interaction using hand tracking
Designing a system that merges music, body movement, and brain activity into a unified experience
Evolving the project architecture through multiple iterations into a robust final system
Demonstrating brain synchronization visually in multiplayer scenarios
Collaborating as a multicultural, international team with members from the United States, Italy, France, and Colombia, bringing diverse perspectives and approaches to the project
Maintaining strong team collaboration and interpersonal harmony throughout the project, enabling effective communication, smooth coordination, and a positive creative workflow

What we learned

We learned how to design and implement real-time interactive systems that combine hardware and software. This included audio analysis, signal processing, EEG data handling, and immersive interaction design.

Equally important, we learned the value of adaptability—being willing to pivot tools, architectures, and approaches when initial ideas proved impractical.

What's next for Between Us

Future plans include improving gesture recognition, enhancing EEG-driven interactions, and expanding multiplayer collaboration. We also want to explore machine learning for pattern recognition in brain and movement data, as well as richer visual effects and more expressive sound synthesis.

Ultimately, we aim to continue developing Between Us as a platform for shared, human-centered audiovisual experiences.