Jamfusion

AI that lets you explore how the world sounds

Comment 1

Jamfusion

Inspiration

Music is one of the most powerful ways culture lives and evolves. But most digital tools treat culture as static: you pick a preset genre or sample pack and move on. We wanted to reimagine this: what if you could see, hear, and interact with culture as a living network, remixing traditions, instruments, and moods in real time?

What it does

Jamfusion is an AI-powered collaborative music creation platform that brings global musical traditions to life through an interactive flow diagram.

Key Features:

  • Interactive Musical Flow Diagram: Build your composition by connecting instruments, genres, and song sections in a visual graph interface
  • LLM-Powered Recommendations: Gemini AI analyzes your composition and suggests culturally-appropriate instruments from 71+ global traditions (Latin, Afrobeat, Brazilian, J-pop, Chinese, Indian, Middle Eastern, Caribbean, Spanish, Electronic, and Hip-Hop)
  • Speech-to-Graph Creation: Speak naturally to build your song structure - "Add drums and bass, then connect them to the verse"
  • AI Producer Voice (via ElevenLabs): A calm producer voice listens to your project and offers culturally-aware feedback in real time:
    • "Your Hip-Hop beat feels strong. Try layering a djembe for an Afrobeat vibe."
    • "This would sound great with a guzheng melody - the Chinese pentatonic scale adds unexpected beauty."
  • Cross-Cultural Fusion: Instead of working inside genre silos, Jamfusion encourages cultural blending with intelligent suggestions:
    • Hip-Hop + Afrobeat djembe
    • J-pop + Chinese guzheng
    • Latin percussion + Electronic synths
  • Drag-and-Drop Workflow: Drag recommended instruments directly onto your canvas and watch the AI automatically suggest connections
  • Smart Edge Generation: AI creates meaningful relationships between elements, showing how different parts of your composition interact

Together, this makes culture feel alive, explorable, and remixable in the digital age.

How we built it

Frontend:

  • React + Next.js for the application framework
  • React Flow for the interactive graph editor and node-based composition
  • TailwindCSS for responsive, modern UI design
  • TypeScript for type-safe component development

Backend:

  • Python + FastAPI for high-performance REST API
  • Pydantic for data validation and serialization
  • Google Gemini 2.0 Flash for LLM-powered recommendations and graph reasoning
  • Structured LLM prompts that process user input and generate graph commands (createNode, connectNodes, deleteById)

AI & Voice:

  • Gemini LLM: Generates intelligent, context-aware instrument recommendations with cultural explanations
  • ElevenLabs API: Converts AI producer feedback into real-time voice guidance with natural, encouraging tone
  • Cultural Knowledge Database: 71+ instruments categorized by culture, genre, and musical type

Music Generation:

  • Integration with the ElevenLabs Music API for generating audio previews
  • Support for multiple instrument types: drums, basslines, melodies, synths, vocals, FX, and chords

Challenges we ran into

  1. Cultural Sensitivity: Designing AI prompts that celebrate culture without flattening it into stereotypes. We solved this by:

    • Researching authentic instrument uses and cultural contexts
    • Writing detailed prompts that emphasize cross-cultural blending and educational value
    • Having the AI explain why each recommendation fits musically and culturally

  2. Graph Auto-Layout: Keeping the knowledge graph readable while handling incremental updates was harder than expected. We implemented:

    • Automatic edge recalculation based on musical compatibility
    • Two modes: "discovery" (auto-generated edges) and "structure" (LLM-directed edges)
    • Smart positioning for dragged nodes

  3. Real-time LLM Integration: Making the system fast enough so that recommendations + graph updates felt live:

    • Implemented 1.5-second debouncing to reduce API calls
    • Used Gemini 2.0 Flash (fast, efficient model)
    • Designed lightweight prompts that return structured JSON
    • Added loading states and error handling for smooth UX

  4. Speech-to-Graph Natural Language Processing: Converting casual speech like "add drums and bass" into precise graph operations required:

    • Careful prompt engineering to extract structured commands
    • Command dispatcher pattern to execute multi-step operations
    • Context-aware updates that understand incremental changes

Accomplishments that we're proud of

  1. Built a living cultural map of music — not just a static library, but an AI-powered recommendation engine that suggests creative cross-cultural fusions
  2. Created an AI Producer persona that feels like a collaborative cultural mentor, offering specific, actionable, and educational feedback
  3. Demoed real-time cultural fusion: Users can drag instruments like "Afrobeat djembe" into a Hip-Hop project and instantly get explanations like:
    • "Adds authentic West African polyrhythmic depth to hip-hop grooves. The 'talking' quality of djembe creates conversational rhythms that blend perfectly with modern beats."
  4. Seamless speech-to-graph creation that understands natural commands and builds complex musical structures
  5. Intelligent edge generation that automatically shows relationships between musical elements

What we learned

  1. Culture thrives when it's explored, remixed, and shared — and tech can make this process playful and accessible
  2. AI isn't just about automation; it can act as a cultural bridge, helping people discover instruments and traditions they've never heard before
  3. Designing AI for creativity is less about correctness and more about inspiration + expression
  4. Voice feedback transforms the experience: Hearing an encouraging producer voice makes the tool feel collaborative rather than mechanical
  5. LLM prompt design is critical: The quality of recommendations depends heavily on how well we structure the prompt, provide context, and request specific output formats
  6. Graph-based music creation is intuitive: Musicians naturally think in terms of connections (verse → chorus, drums + bass, etc.)

What's next for Jamfusion

  1. Community Fusion Library: Let users publish their cultural fusion graphs as templates others can remix and build upon
  2. Global Mode: Explore a world map to discover music traditions by region, clicking on countries to see their instruments and genres
  3. Deeper Music Generation:
    • Expand from short instrument previews to full track exports with AI-generated backing
    • Real-time audio playback as you build the graph
    • Export to DAW formats (MIDI, stems)
  4. Collaborative Multiplayer: Multiple users working on the same graph in real-time, building songs together
  5. Cultural Context Cards: Rich multimedia content explaining the history and cultural significance of each instrument
  6. Advanced Recommendations:
    • Suggest specific chord progressions based on cultural scales (e.g., maqam, raga, pentatonic)
    • Rhythm pattern recommendations that match cultural traditions
    • Dynamic recommendations that evolve as your composition grows
  7. Mobile App: Touch-optimized graph creation and voice-first workflow for on-the-go music making

Built With

  • Frontend: React, Next.js, React Flow, TypeScript, TailwindCSS
  • Backend: Python, FastAPI, Pydantic
  • AI/LLM: Google Gemini 2.0 Flash (graph reasoning + recommendations), OpenAI (graph commands)
  • Voice: ElevenLabs API (AI producer voice feedback)
  • Music: ElevenLabs API
  • Data: Custom cultural instruments database with 71+ instruments from 10+ global traditions

Built With

Share this project:

Updates