WhatDaDogDoing

Inspiration

The Huawei's "Fetching Reality" challenge asked for a 3D dog made from a single photo, that lives in your real room. So I wanted it to feel alive and realistic as much as possible.

What it does

You take a photo of a dog, and about 40 seconds later a rigged, furry 3D version of it shows up in your room through your phone's camera. It wanders, sniffs, sits and lies down on its own, and walks around your furniture. It comes when you tap, eats treats you drop, fetches a ball you throw, and leans in when you pet it on screen. It runs in the browser at whatdadogdoing.vercel.app, with full AR on Android and a camera-and-gyro view on iPhone, and several people can each have their own dog and room at once.

How I built it

A Python server turns the photo into a 3D dog with a Vision Transformer model, fits a dog skeleton inside the mesh, and grows 156K strands of fur over it. Sixty times a second, it picks the dog's next move from 41 minutes of real dog motion capture (motion matching), with smooth transitions and paws locked to the floor. It plans paths around obstacles mapped from the phone's AR room scan and depth sensor. The TypeScript and three.js client, hosted on Vercel, draws the dog with fur physics, real-object occlusion and shadows, and talks to the laptop over WebSockets through a Cloudflare tunnel.

Challenges I ran into

The big challenge is I had no NVIDIA GPU, so everything had to run on a laptop CPU and a phone's GPU, and the dog's rendering could never wait on the ML. Phone browsers were the hardest part: 500K fur strands crashed tabs, and Android's AR mode made the fur see-through over the camera. Also Gaussian Splatting and other 3D image processing technique does not apply for one-shot generation. Testing with an Android phone, the phone's rough depth map also kept cutting off the dog's paws until I stopped the floor from hiding the dog. Petting by tracking your hand in front of the camera was too unstable to ship, so I switched to petting by touching the screen.

Accomplishments that I'm proud of

I realize proud that my dog models moves like a real one, because every step comes from real dog motion capture rather than hand-made animation, and its paws don't slide. The fun thing is, you can go from a photo of your own dog to that dog walking around your living room all takes under a minute, all on a laptop. Anyone can open the link from any network, and several guests can play with their own dogs at the same time.

What I learned

Realistic motion comes from real data plus small details, like locking the feet and speeding up small dogs' steps so they don't look like slow motion. Phone browsers have hard limits on memory, antialiasing and depth quality, and iPhone Safari has no web AR at all, and you only find these on a real device. Visual bugs got fixed fastest when I measured them: I traced the missing paws by noticing their edges lined up with the depth map's pixel grid.

What's next for WhatDaDogDoing

I want the app to remember your room between visits, so your dog is waiting where you left it. I also want friends' dogs to meet and play in the same room. After that, I'd like a more photorealistic coat, petting with your real hand once hand tracking is reliable, and moving the server off the laptop so the dog is always online.