Inspiration

Making a digital double of yourself usually takes a multi-camera rig or a 3-D scanner. We wanted to see how close we could get with what everyone already carries: one phone, two short videos, and a single consumer GPU. Our class project (EEL4403) started from Gaussian Shell Maps (Abdal et al., CVPR 2024), a recent avatar representation. We kept asking one question: could we turn it into something that takes me as input and gives back an avatar I can make dance?

What it does

Record a slow turn in place (clip A) and a few seconds of free movement (clip B).
Track: the pipeline cuts the person out of every frame and fits a 3-D body model (SMPL) to each one. There are no markers and no special suit.
Learn: about 100,000 Gaussian splats on five body-shaped shells learn the person's appearance from clip A.
Move it: the avatar can
- copy clip B, a video it never trained on;
- spin 360°;
- perform moves it was never shown, such as waving both hands at once, squats, jumping jacks and marching, typed in as joint angles.

How we built it

Representation. GSM wraps the SMPL body in $S = 5$ offset shells. Each shell carries a texture of 3-D Gaussians (colour, opacity, scale, rotation). When the body moves, linear blend skinning carries the shells, and the Gaussians ride along:

$$ \mathbf{x}' = \sum_{k} w_k(\mathbf{x})\, \big(R_k(\theta)\,\mathbf{x} + \mathbf{t}_k(\theta)\big) $$

Per-person fitting. GSM is a generative model. We repurposed its shell representation for fitting a single person: one shared texture is optimised against many frames at once, and each frame is rendered under its own pose.

Pose from scratch. For a phone video there are no given poses, so we fit them. For each frame $t$ we optimise the SMPL pose $\theta_t$ and translation $\mathbf{t}_t$, plus one shared shape $\beta$, by minimising

$$ \mathcal{L}t = \underbrace{1 - \mathrm{IoU}{\text{soft}}\big(\hat M(\theta_t,\beta),\, M_t\big)}{\text{silhouette}} + w{kp}\,\frac{\sum_j c_j \,\lVert \pi(J_j) - k_j \rVert}{S\sum_j c_j} + \lambda_s \lVert \theta_t - \theta_{t-1} \rVert^2 + \lambda_a \lVert \theta_t - \theta_0 \rVert^2 . $$

The terms:

Silhouette: $\hat M$ is a differentiable soft silhouette from PyTorch3D, and $M_t$ is the Mask R-CNN mask.
Keypoints: $k_j$ are the Keypoint R-CNN joints, weighted by their confidences $c_j$.
Priors: the last two terms keep the motion smooth and plausible.
Initialisation: each frame starts from a constant-velocity extrapolation of the turn.

Measuring honestly. On the PeopleSnapshot benchmark we lock a train / validation / test split before training. We also report PSNR as the mean of exact re-composites over black and over white:

$$ \mathrm{PSNR} = 10 \log_{10} \frac{1}{\mathrm{MSE}} $$

This keeps the background colour from flattering the score.

on held-out frames	fg PSNR	PSNR (black/white bg)
GSM shells, texture fitted (baseline)	25.25 dB	26.48 dB
+ our per-frame pose correction	25.91 dB	27.52 dB

On validation, the gain is about 90× the seed-to-seed noise we measured (0.014 dB). The benchmark uses the dataset's calibrated camera and body shape, so it does not score the phone path.

Challenges we ran into

A 2023 research stack on a 2026 machine. PyTorch 1.13 with CUDA 11.7 needed exact pins (mkl<2024.1, setuptools<70), plus CUDA extensions built in an isolated environment.
The wrong body. Our first fit capped at 19 dB because the body shape didn't match the person. Switching to the dataset's consensus shape fixed it.
A metric that flattered us. Scoring over a grey background hid halo errors. We moved to the black/white re-composite.
Half-pixel masks. Nearest-neighbour downsampling shifted masks by about 0.5 px. Area-averaging fixed the registration.
Front or back? A person seen from behind looks a lot like a person seen from the front. During our own capture the tracker locked onto the wrong side halfway through a turn. Our left/right "swap check" was reinforcing the mistake, so we made it optional.
Licences. The benchmark's licence forbids redistribution, so every public image in this project is of us, captured on our own phone.

Accomplishments that we're proud of

An end-to-end pipeline that goes from two phone videos to a moving avatar in about 20 minutes on a single 12 GB GPU.
A measured, reproducible +1.04 dB on held-out benchmark frames from pose correction alone, with no extra training data.
Making our own avatar do something we never filmed: waving both hands at once.

What we learned

Most of the quality comes from geometry, not appearance. Better poses and body shape beat a bigger texture or more Gaussians, both of which we tried.
Pick the metric before you look at the results, and make sure it can't be gamed by the background.
Monocular tracking is mostly about ambiguity, such as front versus back and left versus right. Priors and good initialisation matter as much as the loss.

What's next

A personal body shape from a few more photos.
Camera calibration from a printed checkerboard.
Driving the avatar from motion-capture libraries, not just typed-in joint angles.