Project Title: Buu AI - Game Maker

Buu AI - Game Maker is a revolutionary 3D real-time game engine powered by Gemini, Three.js, and multiple 3D generative AI models. It democratizes game development by enabling non-technical users to describe game logic, level designs, set dressings, ideas, and tips in natural language. Gemini interprets these inputs to automate scripting, asset placement, real-time 3D generation, procedural rigging, mesh collider placement, text-to-animation conversion, Mixamo-compatible animations, and seamless public sharing via unique links. Users can iterate instantly, turning concepts like "a cyberpunk city chase with flying drones and destructible environments" into playable 3D games without coding expertise.

Inspiration

Witnessing the frustration of aspiring game creators—artists, writers, educators—barred by steep technical barriers in tools like Unity or Unreal Engine inspired Buu AI. During a 2024 indie game jam, I saw non-coders abandon ideas due to scripting hurdles. This sparked my vision: an AI-orchestrated engine where imagination flows directly into interactive 3D worlds. Influenced by no-code platforms like Bubble and AI advancements in Stable Diffusion for 2D, I aimed to extend this to 3D gaming, empowering 100 million potential creators worldwide to prototype AAA-quality experiences in minutes, fostering innovation in education, therapy, and entertainment.

What I Learned

Building Buu AI deepened my expertise in multimodal AI integration: mastering Gemini's natural language understanding for parsing complex game descriptions into executable logic trees, including conditional branching for quests and AI behaviors. I gained proficiency in Three.js for WebGL-optimized rendering, handling dynamic lighting, shadows, and physics simulations at 60 FPS on browsers. Explored 3D GenAI models like Shape-E and Point-E for procedural mesh generation from text prompts, plus rigging techniques using Blender APIs via JavaScript wrappers. Learned optimization for real-time performance, such as LOD (Level of Detail) systems and Web Workers for parallel asset loading. Discovered ethical AI considerations, implementing bias checks in generated assets to ensure diverse, inclusive representations. Overall, reinforced scalable architecture design for cloud-based AI inference, reducing latency from 5s to under 1s per generation cycle.

How I Built It

Started with a Three.js core for the 3D canvas, initializing a scene with camera, renderer, and physics via Ammo.js for realistic collisions. Integrated Gemini API as the brain: users input descriptions (e.g., "Create a forest level with roaming wolves, collectible berries, and a hidden cave puzzle"), which Gemini tokenizes into JSON-structured commands—e.g., {logic: "AI patrol paths", assets: ["wolf model", "berry props"], animations: "wolf idle/walk"}.

For asset generation: Leveraged multiple 3D GenAI models (e.g., TripoSR for text-to-3D meshes, LGM for UV texturing) via API calls, procedurally rigging skeletons with auto-IK solvers. Implemented text-to-animation using a custom pipeline: Gemini generates motion descriptions, converted to BVH files compatible with Mixamo's retargeting, then imported into Three.js with GLTF loaders.

Scripting automation: Gemini compiles pseudocode into JavaScript functions (e.g., event listeners for player interactions), injecting them into a runtime ECS (Entity-Component-System) framework built on Bitecs for efficiency. Added mesh colliders via Three.js's Raycaster and BoundingBox helpers, auto-placed based on generated geometry.

Public sharing: Exported scenes to GLB format, hosted on Vercel with query params for user sessions; generated shareable links with embedded previews using WebXR for VR compatibility.

Tech stack: JavaScript/TypeScript, Gemini 1.5 Pro, Three.js r165, Ammo.js, GLTFLoader, multiple GenAI APIs (via Hugging Face endpoints), Node.js backend for orchestration. Total build: 48 hours, 2k+ LOC, tested on Chrome/Firefox with 10 user prototypes.

Challenges I Faced

Latency in 3D generation: Initial GenAI calls took 10-15s, disrupting real-time flow; optimized by batching prompts, using lower-res models for previews, and caching common assets—reduced to 2s average.

Integration mismatches: Gemini's outputs varied in structure; built a validation layer with JSON Schema to retry malformed responses, achieving 95% parse success.

Cross-browser compatibility: Three.js physics glitched on mobile; debugged with WebGL extensions, added fallback to Cannon.js, and throttled frame rates for low-end devices.

Animation syncing: Text-to-animation often produced unnatural rigs; fine-tuned with procedural adjustments (e.g., auto-weight painting) and user feedback loops for refinements.

Scalability: High API costs for frequent generations; implemented rate limiting and local ML fallbacks using ONNX Runtime for basic shapes.

Security: Prevented prompt injections by sanitizing inputs; ensured shared games are sandboxed to avoid malicious scripts.

Overcame these through iterative testing with 5 beta users, turning obstacles into features like "AI debug mode" for visualizing logic flows. Buu AI now stands as a robust, user-centric tool ready to transform game creation.