Band Creator

This is a brand new feature for our in-development music rhythm game Jamsesh

it is being bult on Horizon OS for the Meta Quest!

Inspiration Every band has an identity—a logo on the kick drum, custom guitar skins, gig posters that scream "this is us." But in VR, creating custom assets typically requires leaving the headset, using external design tools, and dealing with file uploads. We wanted to break down that barrier. What if you could sketch your band's logo on a napkin during rehearsal, snap a photo without leaving VR, and watch AI transform it into professional assets that appear on your instruments instantly? Band Creator aims to bridge the physical and virtual worlds, letting creativity flow naturally from paper to stage. What it does Band Creator is our vision for using Quest's passthrough camera and AI to transform hand-drawn sketches into custom band assets: Planned Workflow:

Sketch on Paper - Players draw their band logo or design on real paper Capture in VR - Using Quest's passthrough camera, snap a photo without removing the headset AI Transformation - Llama image-to-image AI processes the sketch, enhancing and stylizing it Apply to Assets - Generated designs appear on kick drums, guitar skins, posters, and note highway glyphs

Current Status: We're in early development, focusing on proving the core technical feasibility of the camera-to-AI-to-asset pipeline within the competition timeframe. How we built it Technical Foundation:

Unity 2022.3 LTS with Meta XR SDK v81+ Meta Camera Passthrough API for sketch capture OpenAI integration for image processing Planning custom texture pipeline for asset application

Development Approach: Week 1: Research & Planning

Studied Meta's Camera Passthrough Access documentation Researched Llama image-to-image API capabilities Defined MVP feature set realistic for competition timeline Created technical architecture document Set up Unity project with Meta XR SDKs

Week 2: Camera Integration (In Progress)

Implementing passthrough camera access Building capture UI with framing guidelines Testing image quality and preprocessing needs Prototyping capture workflow UX

Week 3-4: Planned Work

Llama AI integration and testing Basic asset application (kick drum logo) Demo video production Polish and bug fixes

What We've Completed:

Project setup with Meta XR SDK Technical research and feasibility validation Design documentation for full feature Competition submission preparation

What's In Progress:

Camera passthrough implementation AI integration prototyping Basic UI framework

Challenges we ran into Time Constraints: With only 33 days for the competition and a small team, we had to be realistic about scope. We're focusing on proving the core concept rather than building a fully-featured system. Technical Complexity: This is our first time working with:

Meta's Camera Passthrough API Llama AI integration in Unity Real-time texture generation and application

Each component is new territory, requiring research and experimentation. Learning Curve: Understanding the quality limitations of Quest's passthrough camera and how to preprocess images for AI has been a learning process. We're discovering what's technically feasible versus what we initially envisioned. Resource Management: Balancing competition work with our main product development roadmap. We're treating this as an accelerated R&D sprint that will inform future features. Accomplishments that we're proud of Innovative Concept: We identified a genuinely novel interaction pattern—using real-world paper sketches as VR content creation input. Even in prototype form, this feels magical and showcases what's possible when you combine physical and digital creativity. Technical Validation: Through research and early prototyping, we've validated that the core technical approach is feasible. The camera quality is sufficient, Llama can process sketches effectively, and Unity can handle dynamic texture application. Rapid Learning: We've quickly ramped up on Meta's latest APIs and AI integration, knowledge that will benefit our broader product development beyond this competition. Clear Vision: Even though we're early in development, we have a crystal-clear vision of what this feature should be and how it fits into our larger social karaoke platform. What we learned Camera as Creative Input: Passthrough isn't just for mixed reality gameplay—it's a powerful input device for bringing real-world objects into VR. This opens creative workflows we hadn't considered before. Prototype Fast, Validate Faster: Competition constraints forced us to focus ruthlessly on the core value proposition. Rather than building everything, we're proving the most important piece: can we make the camera-to-AI-to-asset pipeline work? AI Integration Complexity: Working with AI APIs in real-time VR applications has unique challenges around latency, error handling, and user experience. We're learning how to make AI feel responsive and reliable. Realistic Scoping: We learned to be honest about what's achievable in a short timeframe. It's better to have a solid, working prototype of one feature than a broken implementation of many features. Documentation is Everything: Meta's developer documentation and sample projects have been invaluable. The SharedSpaces and Camera API samples saved us significant R&D time. What's next for Band Creator Immediate Next Steps (Competition Completion):

Complete camera capture implementation Integrate Llama AI processing Apply generated logo to kick drum asset Create demo video showing end-to-end workflow Submit to competition by December 9th deadline

Post-Competition (If Successful):

Expand to additional asset types (guitar skins, posters) Add multiplayer synchronization (show designs to other players) Implement design gallery and variation selection Polish UI/UX based on user testing Integrate into main Jamsesh product

Long-Term Vision: Band Creator is a proof-of-concept for a much larger creative system. If this prototype succeeds, we envision:

Full band identity creation suite Collaborative design tools for band members AI-powered design suggestions based on music style Marketplace for player-created designs Physical merchandise integration

Learning Opportunity: Regardless of competition outcome, this sprint is teaching us valuable lessons about rapid prototyping, AI integration, and creative VR tools that will inform Jamsesh's future development.

This is our first major technical challenge in VR development, and we're using the competition as a forcing function to learn fast and prove concepts that will make Jamsesh more creative and social.