LA29 Pixel Hunters: Puppet anything with everyday objects

Video: https://youtu.be/nPcuvOGmx_s

Inspiration

As kids, imagination could turn anything on the table into part of a living world. Scissors became an alligator, two fingers became a walking character, and a paper airplane became a jet. We wanted to bring that form of embodied play into interactive 3D. Instead of animating with timelines, keyframes, or specialized controllers, what if people could manipulate the ordinary objects already around them and use those motions to puppeteer virtual characters?

What it does

PuppetAnything is a mobile and web app that turns everyday objects into controllers for virtual characters. A user points their phone at an object and performs with it naturally. PuppetAnything tracks the object’s position, orientation, and simple articulation, then maps that performance onto a corresponding 3D asset.

For example:

• opening and closing scissors controls an alligator’s mouth
• walking two fingers controls a character
• flying and banking a paper airplane controls a virtual jet

The virtual assets are generated with Tripo and placed inside an interactive 3D environment powered by World Labs.

Interactive world: The world is not static. Instead, objects come to life when clicked or when the character interacts with them. Think of it as Night at the Museum.

How we built it

We built PuppetAnything as a real-time mobile vision and 3D interaction pipeline. On the phone, EdgeTAM tracks the selected object at 30 frames per second. We combine its persistent segmentation with RGB-D information and device camera poses to recover the object’s motion in 3D. For articulated objects, we analyze tracked point trajectories during a brief calibration motion. Points that move together are grouped into rigid components, allowing us to estimate simple one-joint or two-joint mechanisms. Their relative motion becomes normalized control signals, such as mouth openness or neck rotation.

We then map:

• the physical object’s root motion to the virtual asset’s root transform the inferred articulation values to prepared joints in the virtual asset
• Tripo provides the generated 3D characters and vehicles, while World Labs provides the interactive world in which those assets are puppeteered.

Challenges we ran into

Mapping Any Object to a 3D Character: We wanted to support non-humanoid characters and arbitrary control objects. For example, a child might use a stick to control a flying bird. We used two approaches:

• Semi-automatic: Match tracked motion to the target model’s bone structure.
• Automatic: Use the Gemini API to determine which bones should be controlled from the input and target poses. b. Tracking Articulated Objects

A segmentation mask does not reveal separate moving parts, such as the two arms of scissors. We solve this by grouping 3D point trajectories based on their motion, allowing us to identify rigid parts and simple joints without a category-specific pose model.

Accomplishments that we're proud of

We are proud that PuppetAnything turns an ordinary object into a responsive 3D controller without requiring the user to create a rig, place keyframes, or wear motion-capture equipment. We built an end-to-end interaction in which:

• object tracking runs live on-device
• simple articulated motion is recovered from a whole-object track
• physical motion is transferred to generated 3D assets
• the same object controls both the character’s movement and its expressive joints
• the resulting performance takes place inside an interactive generated world

Most importantly, the interaction remains understandable immediately. Move the scissors and the alligator moves. Open them and its mouth opens.

What we learned

We learned that everyday-object puppeteering does not require recovering a complete semantic skeleton. For many playful interactions, a much smaller representation is enough:

• one root transform
• none or two articulated components
• a few normalized control values

Built With

• three.js
• tripoai
• worldlabs