Inspiration

I've always found building with blocks one of the most natural things to do in VR. Mixed reality takes it further, because now it happens on your own table. And with what the headsets offer now, camera access, live depth, solid hand tracking and eye gaze, I could finally make the blocks feel like they're in the room with you.

What it does

Blockworks is a building game that blends into your room. The blocks react to your real hands, hide behind your furniture and collide with the room itself. They have weight, so they tip, slide and tumble off the table.

In Blueprint you're guided through 20 handmade builds. In Sandbox the whole room is your canvas.

How I built it

The goal was to make the blocks look like they belong in your room. I spent the first week trying out everything the headset offers to see what could get me there, and after that most of the work went into the lighting.

The headset's camera is sampled constantly into a cubemap of your room, and the parts it hasn't seen yet are filled in from the parts it has. That cubemap is blurred, then studio highlights are composited on top, so every block takes on the colours of your room and still gets a crisp highlight. Getting that mix to look right is where most of my time went.

Live depth hides the blocks behind your hands and furniture. It also tells the game when your hand has gone behind something, so that hand is ignored instead of dropping your block. Every finger is a physics body, so a flick sends a block flying. You pick a block up with a pinch, aiming with your hand or, where the headset tracks them, with your eyes.

Challenges

The headset is a small computer, and passthrough, depth and hand tracking all take their share before the game gets anything. So every feature had to earn its place. I tried several occlusion methods and post-processing effects that looked good and cost too much, and cut them. What stayed was measured on the device, with the heavy work spread over several frames, until the whole thing ran comfortably.

The hands took just as long to get right. Fingers that can push blocks also push away the block you're trying to pick up, so a hand has to stop being solid the moment it grabs and turn solid again when it lets go. Making the pinch light enough that nobody has to think about it took a lot of small adjustments.

The camera only sees a narrow slice in front of you, so the reflections fill in as you look around, and the game works out the parts of the room it hasn't seen yet.

What's next

Mostly I want to keep iterating and polishing, and to get Blockworks to its best on Meta VR Glasses, where eyes and hands are the whole way you play. The feature I'm most curious about is shared play, two people building on the same table in the same room. After that come more custom sets, seasonal packs, and new blocks and colours.

Built With

  • depth-api
  • eye-tracking
  • hand-tracking
  • passthrough
  • passthrough-camera-api
  • scene-api
  • unity
  • urp
Share this project:

Updates

Submission history