Inspiration
T’ALAT was inspired by our families. Lizbeth thought of her little cousin, who once had difficulty expressing herself and has made progress over time. Carlos thought of his younger brother, who has autism, and how everyone experiences and expresses the world differently. Aldo thought of his uncle, who has cerebral palsy, and the barriers he may face when playing a traditional instrument.
We also met Vane at HOGA. She arrived feeling very sad and had not wanted to eat. When she tried T’ALAT, her facial expressions made the piano play, and we saw her smile. Her experience showed us how music can help people express themselves and connect with others.
What it does
T’ALAT uses artificial intelligence to recognize facial gestures and turn them into musical notes. This gives people who face barriers when using traditional instruments another way to create music and express themselves.
How we built it
We combined programming, artificial intelligence, and electronics. We used a camera and Python tools such as OpenCV and MediaPipe to recognize facial gestures. A Raspberry Pi processes the information, and relays connect the system to an adapted musical keyboard so the gestures can produce sounds. We shared the work across programming, components and connections, documentation, and design.
Challenges we ran into
We had to improve how accurately the system recognized gestures, adjust it so that unintended movements would not play notes, and make the camera, software, Raspberry Pi, relays, and keyboard work together. Feedback from users and specialists helped us improve the instructions, gesture detection, and system integration.
Accomplishments that we’re proud of
We tested T’ALAT with 35 users at HOGA, DIF Apodaca, Lazos de Apodaca, the Apodaca Municipal Robotics School, and Gregor Johann Mendel Educational Center. Five therapists, six psychologists, and four teachers also shared feedback that helped us improve it.
We are especially proud of the moment Vane smiled while making music with her expressions. It reminded us that our work can create meaningful experiences for people.
What we learned
We learned that inclusive technology starts by listening to people and understanding their needs. Testing, receiving feedback, and making changes are important parts of building something useful. We also learned that people express themselves in different ways, and technology can help make more opportunities possible.
What’s next for T’ALAT
We want to keep improving T’ALAT with the feedback we receive and help more people try it in schools, support centers, and other spaces. Our next steps are to make the system easier to use, improve its gesture recognition, and keep creating opportunities for people to participate in music in their own way.