BlindMelodies

Workflow

While brainstorming project ideas, we wanted to create something that could make a real impact and provide accessibility to people with disabilities. After considering different approaches, we decided to focus on helping blind individuals learn to play the guitar. The idea came from our passion for music and the realization that many resources for learning instruments are visual-centric, leaving a gap for those who are blind or visually impaired. Thus, BlindMelodies was born—a project that combines music with voice technology to provide an interactive, accessible guitar learning experience.

What it does

BlindMelodies is a web application designed to teach blind individuals how to play guitar using voice input and text-to-speech technology. Users can interact with the app by saying which chord or song they want to learn, and the app will respond with the necessary information, including guitar chords and song details. The app leverages the Gemini API to process the voice input and provide accurate responses. It also keeps track of the user's progress and provides continuous feedback.

How we built it

We built BlindMelodies using a combination of modern web technologies:

Frontend: React and Vite were used to create a responsive user interface. We also utilized Tailwind CSS for styling to ensure that the app is clean and easy to navigate.

Voice Interaction: The app uses text-to-speech (TTS) for guiding users and listens for voice commands to understand whether the user wants to learn a chord or a song. API Integration: The app integrates with the Gemini API to process the voice inputs and provide relevant guitar chords or song details. State Management: The app keeps track of the user’s requests and provides personalized responses based on their previous interactions. 3D Interaction: Spline was used to create interactive 3D models, providing a visually engaging experience that complements the app's learning features

Challenges we ran into

Voice Recognition Accuracy: Handling requests from the user and ensuring that the app accurately recognizes and responds to user requests was challenging.

Real-Time Interaction: Providing real-time responses for learning guitar proved to be tricky, especially when managing voice input and TTS in a seamless, interactive manner.

Accessibility: Ensuring the app was fully accessible to blind users required additional testing and adjustments. We had to make sure that all audio feedback was clear and informative, providing the right context for each learning session.

Accomplishments that we're proud of

Voice-Driven Interface: The successful integration of voice interaction is one of the main accomplishments. We managed to create a seamless voice-driven user experience that’s both intuitive and helpful.

User-Friendly: The app is simple and easy to use, even for those with little to no experience with technology. The voice commands guide users throughout the entire learning process.

We ensured that the page operates autonomously, initiating automatically upon pressing the spacebar, eliminating the need for any manual button interactions.

Impact: We’re proud that our project can help blind individuals learn an instrument they might otherwise be excluded from due to the visual nature of traditional learning resources.

What we learned

Voice Technology: We gained a deeper understanding of integrating voice recognition and text-to-speech functionality into a web application.
Accessibility Best Practices: Designing for accessibility was a key takeaway, and we now have a greater appreciation for how to build more inclusive and user-friendly applications.
React and Tailwind CSS: We learned a lot about using React with Vite and Tailwind CSS, which helped us develop a fast, responsive, and visually appealing interface.

What's next for BlindMelodies

Adding More Features: We plan to add more songs and chords, as well as enhance the app’s ability to provide more personalized learning experiences based on user progress.
Expanding Voice Interaction: Further improving the voice recognition to understand more complex commands will be a focus, making the learning process more dynamic.
Mobile Compatibility: Ensuring the app works seamlessly across mobile devices is another next step, as this will make the app more accessible to a wider audience.
Collaboration with Music Institutions: We hope to collaborate with music schools and organizations to help further refine and distribute BlindMelodies, bringing music education to more blind individuals around the world.