DAWGS: Animated with Gaussian Splat

Ball fetched
Fetching ball
Menu to add your own dog

Inspiration

We wanted to learn hardware and try a cutting-edge graphics technique, Gaussian splatting. None of us has had a pet, so we decided to build a realistic dog we could actually play with: one that reacts when you pet it and chases a ball when you throw it.

What it does

Upload a photo of any dog and TRELLIS turns it into a 3D Gaussian splat model. Splats keep the photo's fur and lighting, so the dog looks real rather than cartoonish. Your dog joins a roster (a beagle is the default) and appears in a grassy environment, where you play with it using two Arduino joysticks:

Camera joystick: orbit around the dog and tilt the view.
Ball joystick: toss a ball. It bounces and rolls with real physics, and the dog chases it, picks it up in its mouth, and brings it back to you. Moving this joystick also pets the dog, which turns toward you, leans into the scratch, and wags its tail.

No hardware? Keyboard controls do the same things.

How we built it

TRELLIS (via Hugging Face) to convert the 2D image to a 3D splat
Three.js + Spark to render the dog and background in one scene
A procedural skeleton fitted automatically to each dog's shape, which animates the splats on the GPU so any dog can walk, trot, sit, and fetch
Arduino + joysticks streaming input to the browser over Web Serial, with no server in between

Challenges we ran into

Finding a model and starting image that produced a usable 3D dog. Results depended heavily on the photo, and the free GPU quota limited how many tries we got.
Getting a 350k-splat dog to actually move. A splat is a frozen snapshot, so animating it means going from 3D to 4D Gaussians. We built a skeleton that bends the splats in real time.
Making the animations look good. Moving legs and an opening mouth reveal parts the photo never showed, which leaves gaps. Ideally we'd rent cloud GPUs to fill them in.
Getting the dog to follow the ball. The scene had no floor to collide with, so we pulled ground height and obstacles out of the background splats.

Accomplishments that we're proud of

Integrating hardware into the project even though most of us had never worked with hardware before
Getting deep into computer graphics for the first time, on a problem that's only become solvable recently: Gaussian splatting itself is just a few years old, and animating splats is even newer
Turning any dog photo into a 3D dog that plays fetch

What we learned

How to set up an Arduino for the first time, wire joysticks, and stream their input to a web page
How Gaussian splats work and how to render and animate them with Three.js