MuzeAI

Are you interested in playing the piano and producing music? Our tool lets you create your own music and provides an AI Agent that can assist you during your music creation process.

Comment

Inspiration

Our primary interest with this project was music production. We were also very interested in creating software that makes learning new concepts easier and quicker. We drew inspiration from software such as Bosca Ceoil and BeepBox for our project.

What it does

This project makes music production much faster with the use of an AI agent. You have full freedom over the creation of your music, and you can optionally use an AI agent to help modify your music and generate new music. You can edit music in a note grid format similar to other popular music production programs. However, you get a chatbot on the side that can understand your music, modify your music, and create new pieces.

How we built it

React for frontend: the frontend software for the application was built using ReactJS. It includes the piano roll and a real-time audio playback feature. Supabase for backend: The application is backed by Supabase for the database management and user authentication. Gemini to power the AI Music Producer bot: The main program is powered by Gemini API. The API is used to allow the LLM to create, modify, and remove notes from the music track.

Challenges we ran into

Connecting text to Music: Designing a data schema for Gemini to accurately interpret the piano roll data was quite tedious and time consuming. We then had to ensure that the AI was returning valid JSON files for the program to use and play. UI & Audio Sync: We ran into some trouble with managing react states, the visuals, and the audio. For example, there were some ghost audio clips that would continue to play even after the program was paused or fully reset.

Accomplishments that we're proud of

Piano Roll Design: We are proud of building a responsive piano roll interface that is user-friendly and fun to use. Fully interactive AI Assistant: We are proud of getting the Gemini agent to not only generate notes similar to a black-box generator, but to actually understand the notes present in the music file and modify it to the user's liking.

What we learned

Supabase: We learned how to use Supabase to store project files and user authentication data. React: We learned how to use ReactJS to create and manage complex UI and program states. Gemini API: We learned how to make use of Gemini API to prompt LLMs with project specific data.

What's next for Muze.AI

Multiple instruments: We plan on adding multiple new instrument types for the program. Adding more instruments would make the program more fun to use. Midi Keyboard support: We plan on integrating Web MIDI API for users to record music from physical MIDI keyboards. Project sharing: We plan on adding a community page for the users to make their projects public for others to view and use.

Built With

Share this project:

Updates

Submission history