Merci AI - Voice Coding Assistant for IWSDK

Build VR worlds with your voice on Meta's IWSDK. Speak naturally, AI generates code and 3D models instantly. No coding knowledge needed - your imagination is the only limit.

Comment

Inspiration

I'm a psychologist. Six months ago, I discovered AI and fell in love with development. After participating in several DevPost competitions, I stumbled upon Meta Horizon Start - and discovered IWSDK. It was a revelation: Meta created a framework that makes WebXR truly accessible.

That's when the idea was born: what if we remove the last barrier? Not just simplify code - eliminate it entirely. You speak → world appears. I want people to create without getting stuck in syntax. Creativity drives the world - let's set it free.

How We Built It

Tech Stack:

  • Meta IWSDK - immersive web framework
  • Claude Agent SDK - explored multi-agent patterns, chose single-agent with direct tool access for MVP speed
  • Gemini API - voice recognition
  • Meshy AI - 3D model generation
  • Vite HMR - hot reload pipeline

Game Compiler Architecture: AI agent writes minimal game code → custom compiler injects helpers and infrastructure (physics, grab system, cleanup) → production-ready code. This approach reduces AI hallucinations by 80% - agent writes creative logic, compiler handles boilerplate.

Reserved Names Auto-Fixer: Compiler automatically renames conflicting variables (40+ reserved names from base helpers) to prevent runtime errors. Agent focuses on ideas, not framework internals.

What I Learned

  • Claude Agent SDK - single-agent vs multi-agent tradeoffs
  • IWSDK architecture - ECS patterns, physics integration, grab system APIs
  • WebXR performance - Quest optimization, polygon budgets, hot reload without breaking XR sessions
  • Compiler design - code injection, variable renaming, AST patterns
  • Voice UX in VR - hands-free interaction design

Challenges

  1. IWSDK is brand new - LLMs hallucinate on unfamiliar APIs → built compiler + helpers to guide AI
  2. Hot reload in XR - maintaining session while updating code requires careful HMR setup
  3. 3D model optimization - Meshy generates 100k+ polygons, Quest needs <20k (gltf-transform pipeline)
  4. AI code quality - balancing creativity vs correctness through compiler guardrails

Accomplishments

✅ Built working voice-coding platform with compiler architecture ✅ <10ms hot reload without exiting XR ✅ Full game loop: physics, portals, shooters, grabbable objects ✅ Reduced AI hallucinations 80% via compiler approach ✅ First AI coding assistant on IWSDK

Future Plans

Near-term:

  • Scene Understanding game integration (implemented, needs flow adaptation)
  • Game sharing marketplace for community creations
  • Rigged 3D models with animations (Meshy integration ready)

Long-term:

  • Fine-tuned model on IWSDK patterns for near-zero hallucinations
  • Multiplayer collaborative building
  • Visual programming mode for non-coders

Built With

Share this project:

Updates