Inspiration
Music has the power to unite, inspire, and uplift, but access to music education and instruments remains a privilege for many. At OrchestrAir, we believe music should be for everyone—no barriers, no exceptions, regardless of circumstance. Our interactive sandbox empowers individuals, from curious beginners to seasoned musicians, to create and learn music using an intuitive virtual environment, anywhere, anytime. Through this application, users can compose melodies with simple hand gestures, visualize their creations as digital sheet music, and even generate custom album covers to share their art with the world. As proud advocates for the UN Sustainable Development Goals (SDG 4: Quality Education & SDG 10: Reduced Inequalities), we are committed to harnessing technology to foster inclusivity and reduce disparities in music education. OrchestrAir isn’t just an app; it’s a movement to make the joy of music universal.
What it does
OrchestrAir is an interactive and immersive music sandbox that redefines how you create, learn, and experience music. With Melody, your personal AI-powered music tutor, you can explore music theory and receive personalized, real-time assistance with any questions. Play life-like instruments like the piano and drums, hearing realistic note sounds and frequencies as you compose. OrchestrAir enables you to record your compositions in real-time, generating music sheets that can be viewed, edited, and downloaded as PDFs. Additionally, Melody can create custom music sheets based on your prompts, providing tailored songs instantly. To enhance creativity, OrchestrAir generates personalized album cover images inspired by the notes you play, letting you bring your musical vision to life with downloadable visuals.
Create Music with Gestures: Using intuitive gesture-based controls, users can compose melodies, harmonies, and rhythms in real-time, making music creation accessible to everyone—even those with no prior musical experience.
Learn Instruments Anywhere: The platform serves as a virtual music tutor, offering interactive tools to help users explore and learn musical instruments, no matter where they are or what resources they have. Melody the AI Music tutor has comprehensive tutorial pages dedicated to step-by-step teaching of piano, can conversate with you regarding anything music-related, and generate you a custom music sheet based on any given prompt.
Save and Share Creations: OrchestrAir automatically converts your musical compositions into professional-looking music sheets and generates custom album covers, allowing you to showcase your creations to the world.
Inclusive Music Experience: Designed with accessibility at its core, OrchestrAir empowers underprivileged and disabled communities to experience and participate in music creation and education.
Bridge the Gap in Music Education: By integrating cutting-edge technology and a passion for inclusivity, OrchestrAir makes high-quality music education available to anyone, regardless of their background or circumstances.
Whether you're a curious beginner or an expert musician, OrchestrAir is your gateway to a boundless world of music, ready to turn your ideas into melodies.
How we built it
Our app was created using Flask to run the backend server along with a React Frontend. We also leveraged a lot of really helpful libraries and APIs to create our functionality. In the backend, we use openAI for the chatbot and music generation, as well as tensorflow for album cover generations. We also used music21 to create our sheet music and play them out loud. In the frontend we use three.js to create out piano keys, recording button, and the 3D hands.
Challenges we ran into
Matching MediaPipe Coordinates to Canvas
Challenge: Rendering the user's hands in real-time on the canvas with minimal latency was a significant hurdle. MediaPipe's hand-tracking API provides highly accurate results, but translating these coordinates into a smooth and responsive on-screen experience required careful synchronization. Even a slight delay could disrupt the user experience.
Solution: We optimized the frame processing pipeline by reducing redundant computations and limiting the canvas redraw frequency to match the browser's refresh rate. Additionally, we adjusted for discrepancies in coordinate scaling between MediaPipe and the canvas.
Animating Melody the AI Assistant
Challenge: Creating a visually engaging and interactive AI assistant was central to making the project feel dynamic and fun. We wanted Melody to be more than just a chatbot by animating it to "fly" across the page and "flap" its wings like a real bird. Coordinating these animations while ensuring they looked smooth and did not interfere with user interactions proved to be a creative challenge.
Solution: We implemented a sprite animation technique where Melody alternates between three states: "Stationary," "Flight1," and "Flight2." By swapping these images in rapid succession during flight, we achieved a natural flapping effect. To keep performance high, animations were paused when Melody was stationary or not clicked, conserving system resources.
Music Sheet Generation
Challenge: At one point during development, music sheet generation functionality stopped working. This issue stemmed from changes in our codebase, where dependencies were inadvertently mismatched or improperly configured after several commits. Debugging this was tricky due to the complexity of the feature and its integration with external libraries.
Solution: We systematically rolled back commits and reintroduced changes incrementally to isolate the problem. Once identified, we updated our dependencies, restructured parts of the feature, and added automated tests to ensure future changes wouldn’t break this functionality again.
Accomplishments that we're proud of
Melody, your personal music tutor We created a cutting-edge AI-powered tutor and tutorial platform that adapts dynamically to users' learning styles and preferences. The system delivers tailored explanations, interactive problem-solving exercises, and feedback loops, ensuring a highly engaging and effective learning experience. It’s designed to scale across diverse subject areas, empowering learners of all levels to excel.
Responsive real-time 3D hand generation This is probably the most visually appealing part of our project. We began by sending the mediapipe hand coordinates to the front end, but to make the hands look nicer we interpolated additional points to prevent a triangluar looking palm. Then, we had to correctly map the mediapipe coordinates onto the canvas to create an interactive experience with the record button and piano keys. For this we had to bust out some math, solving the linear equations where the input is mediapipe x and y values, and the output is the canvas coordinates. The galaxy appearance in the hands was somewhat of a happy accident during trial and error, but we kept it because it looks cool.
What we learned
We learned a lot creating this project but something that stands out was exploring three.js. Initial efforts were rusty for generating 3D models, but we got more comfortable with it quickly and it ended up being one of the most interesting parts of our project.
We also learned how to use music21 to create sheet music and play it back out. This is something we're really interesting in using for future projects because its so easy to use and such a cool feature.
What's next for OrchestrAir
The future of OrchestrAir involves taking "connectivity" to the next level. Our future visions include a focus on multiplayer, leaderboards, friend's lists etc.
Log in or sign up for Devpost to join the conversation.