MuseAI

We’re in the Art track, building an AI that turns short vocal clips into full songs, blending human emotion with machine creativity.

Comment

Inspiration

Learning to improvise in jazz takes years. Unlike classical music, which emphasizes structured feedback on technique and dynamics, jazz relies on live interaction and mentorship to develop phrasing and creativity, resources that are often costly or inaccessible. Existing apps, such as Ireal Pro and DrumGenius, only offer backing tracks or beats, not meaningful feedback.

MuseAI changes that. It’s an AI music companion that teaches musicians how to respond to motifs in a jazz jam, fostering creative responsiveness, the essence of improvisation. We are hoping to change the learning trajectories of musicians worldwide.

What it does

MuseAI takes a musical phrase—whether from user vocal input, an uploaded recording, or one of our sample motifs—and generates a stylistically coherent response or complete composition. It’s designed for practical musical contexts such as trading, comping, and call and response. MuseAI can produce multiple responses across diverse musical genres—not just jazz—and allows users to define the desired emotional character of the output.

How we built it

Frontend

• Framework: React.js (with Vite + TypeScript)
• UI Library: Mantine (modern, customizable components)
• Animation: Framer Motion (smooth, responsive transitions)
• Hosting: AWS Amplify
• Features: Fully responsive, high-performance UI optimized for speed and accessibility

Backend

• Framework: FastAPI (Python 3.11)
• Deployment: AWS EC2 instance with custom HTTPS protocol
• Architecture: RESTful microservices design, stateless communication
• Scalability: CI/CD ready, loosely coupled service structure for modular scaling

AI / ML

• Model: MusicGen (AI-based music generation)
• Compute: AWS GPU instance (NVIDIA T4)
• Integration: Model served via FastAPI for seamless frontend–backend communication

Challenges we ran into

  • Keeping the generated music faithful to the user’s melodic and rhythmic intent.

  • Balancing real-time responsiveness with high-quality output.

  • Seamlessly integrating the frontend user interface with backend generative systems.

Accomplishments that we're proud of

We built a prototype that generates motif-aware, musically consistent responses from raw voice or instrument input, bridging human creativity and AI intuition.

What we learned

We deepened our understanding of audio processing, music representation learning, prompt engineering, and how to effectively connect the backend and frontend for seamless user interaction.

What's next for MuseAI

  • Real-Time Jam Assistant: Suggests responses live during jam sessions.
  • Improvisation Distillation: Summarizes a full improvisation into a concise, stylistically coherent musical response.
  • Collaboration Platform: Connects learners with peers or teachers for feedback.
  • Style Personalization: Expands genre and artist-style control.

Built With

Share this project:

Updates