Inspiration
Making things with AI still means tabs, prompts and downloads on a flat screen, even for people who build for 3D and VR. I wanted the opposite: sit down, put on a headset, say what you want, and watch it appear in the room at real size. Then keep building with your hands. Meta VR Glasses ship without controllers, so from day one everything had to work with hands, eyes and voice alone.
What it does
Swirlflex surrounds you with a ring of 206 generative models from 31 makers, for images, video, music, voice and 3D. Pinch a card, or just say "a glowing paper lantern shaped like a fox". An agent picks a fitting model, fills in the settings and tells you the exact price before anything is spent. Say "create" and the result appears beside you. Pictures hang in frames and videos play on screens you can anchor to your real wall. Sound plays in space, and 3D models land on your real floor or table (found with hit-testing and plane detection) at real-world size: grab them with one hand, resize them with two. Or say it: "make it bigger", "copy it", "undo".
Then open Worlds: arrange your models into a scene around you and save it. The same world opens in Swirlflex's web scene editor and exports to Godot, Unity, Unreal and Blender with every position and size kept. Press To editor under any model and it lands a few seconds later in the project open on your computer. It arrives game-ready: lighter, standing on the ground, at real size, with collision.
How I built it
Swirlflex runs on WebXR (three.js) with passthrough, anchors, hand tracking and eye gaze, plus Next.js, Postgres and engine plugins for Unity, Unreal, Godot and Blender. I wrote the hands-first input system from scratch:
- Fingertip touch for buttons, with a keyboard that opens within reach of your hands.
- Pinch rays that ignore hand-tracking jitter.
- Look-and-pinch with eye tracking.
- Two-hand scaling.
- A voice layer that knows when you've stopped talking.
The layout measures the headset's field of view and fits itself to it, from Quest 3 down to the VR Glasses' narrower view. All the model cards are drawn in 5 draw calls.
Safety is built in. Every prompt passes a word filter and an AI reviewer, every result is checked before anyone sees it, and accounts that try to make harmful content are paused.
The AI models run through fal.ai, and they're interchangeable suppliers. The spatial interface, the agent, the world builder, the bridge to game engines and the safety system are Swirlflex's own work. Take the models away and you still have a hands-first tool for arranging 3D worlds and sending them to your engine.
What's next
The goal: a whole game world, built from inside by hand and voice.
- Floors, walls, ground and sky.
- Walk inside your world.
- Game rules: physics, scoring, reactions.
- Multiplayer, and Meta VR Glasses in 2027.
Built With
- blender
- claude
- digitalocean
- fal.ai
- gltf
- godot
- meta-quest
- next.js
- node.js
- postgresql
- prisma
- react
- tailwindcss
- three.js
- typescript
- unity
- unreal-engine
- webxr