dog frisbee
dog ball
dog pet
hotdogdog
Cat
dog

Fetch turns a photo of your own dog or cat into a realistic 3D pet that lives on your screen, plays when you want to play, and runs errands across over 1500+ connected apps when you need a hand, autonomously completing tasks while you work.

Inspiration

Your pet doesn't come to work with you, but its photo is probably on your phone. And it never sees what an AI assistant does all day, because nobody does. You type a request, a spinner turns, a paragraph comes back. The work is invisible, and so is the thing doing it.

We kept thinking about the little pets that live in the corner of a coding tool like Codex and Claude Code, and the desktop pets people had in the 2000s. They were small and kind of pointless, but people loved them anyway. What if that pet were yours, rebuilt in 3D from a real photo? And what if it were actually useful, so that when you asked for something it trotted off, did the work in your apps, and came back with it in its mouth?

That became Fetch: a companion that's half Tamagotchi, half agent. Play with it when you need a break. Hand it ANY task when you're busy. Either way it's a whole lot easier than training a real dog to roll over.

What it does

You make the pet from a photo or a drawing. Upload a picture of your dog or cat, take one with the camera, or draw one. Fetch removes the background, rebuilds the animal as a 3D Gaussian splat (a few hundred thousand tiny colored blobs that together look photographic), fits a skeleton to its body, and drops it onto your desk in about a minute. If your pet was sitting in the photo, Fetch first re-poses the same animal standing, with the same markings and eye color. A crayon doodle becomes a realistic animal that keeps the drawing's colors and pose.

It lives on your screen: The desktop app puts the pet on top of everything else you have open. It walks along your taskbar, wanders, naps and sometimes wanders back. Click it and it reacts. Stroke it and it leans in. Right-click it and it'll sit, roll over, play dead, dance, chase a ball or come to your cursor. The rest of the screen stays clickable, because the window only catches the mouse when you're touching the pet.

It plays Press Play and you're in your pet's own 3D room: a bed, a food bowl, a toy chest, a hoop and a tunnel. Walk around, charge up a throw, and it fetches the ball or frisbee and brings it back. Hold food and it begs. Point at the hoop and it runs the course. It has energy, hunger and happiness, and they decide where it goes when you leave it alone. Flip to camera mode and the pet stands in your real room through your phone's camera, stays put as you turn, and lets you take a photo together.

It listens and talks back Say "sit", "roll over", "give me your paw" or "surprise me" and it does it, using ElevenLabs for voice output and transcription. It answers in a short line in its own voice, and its mouth moves with the words. Dogs are eager about it. Cats are not ("Fine. Sitting.").

It runs errands: Type or say something like "find my budget sheet" or "email the standup notes to my team." The agent plans the job as a list of steps, and every step plays as an animation in your pet's own style. A dog searches nose-down and zig-zagging. A cat stalks. While the pet is off-screen working, a small window at the edge of your screen shows it digging through a growing pile of papers. When it's done it comes back with the document in its mouth. It works in Gmail, Google Drive and Google Calendar, and over 1500+ more apps through Composio. If an app connector doesn't exist, Fetch will find the custom MCP server needed to complete your task, connect it and complete the task.

It asks before anything goes out: Sending an email, sharing a file, deleting something, or any action that involves: the pet comes back holding an envelope and an approval card shows the exact recipients, subject and body. Approve with a button, by voice ("send it"), or by holding down on the pet for a second. Edit the draft and it asks again, because what runs is exactly what you approved.

How We Built It

    PHOTO / CAMERA / DRAWING
               |
               v
 Re-pose or "make it real"   (FLUX.1 Kontext)
               |
               v

Background removal + image-to-3D (TRELLIS) ~260k Gaussians, ~25 s | v Orient -> fit skeleton -> skin weights (in the browser) | v Pet bundle saved on our server | +-------------+--------------+ | | | v v v Desktop Play room Camera view overlay (3D room) (your room) | | "find my budget sheet" v Agent loop (Gemini, tool calling) plan -> step -> tool -> result | | | v | Gmail / Drive / Calendar / Composio | | | Approval gate (exact payload, hashed) v Live events (SSE) -> pet animations + edge peek

Generation: Our server runs the heavy models on hosted GPUs through Gradio. TRELLIS turns one image into a Gaussian splat, and FLUX.1 Kontext does the image editing first: standing up a sitting pet, or turning a drawing into a believable animal. Jobs run in the background, and the app polls them with friendly stage names ("Getting Biscuit to stand still for the sculptor").

Rigging a cloud of blobs:A splat has no mesh, no bones and no idea which end is the head, so we measure the anatomy from the splat itself:

Legs: the columns of points touching the ground give us four legs, split front, back, left and right.
Belly line: where the space between the front and back legs fills in.
Head and tail: the head is the dense high mass in front of the shoulders. The tail is whatever sticks out past the rump.

Rendering: We wrote our own WebGL splat renderer on three.js, with skinning on the GPU. Each splat is projected as a 2D Gaussian with the standard 3DGS math. Splats are depth-sorted every frame they move, using each splat's posed position (one matrix row per bone keeps that as cheap as a static sort). A second renderer draws soft "plush" versions of pets by raymarching signed distance fields on the same skeleton, so every trick works in both looks.

The agent: A Fastify server runs the agent loop on Gemini with function calling, and can also run on Claude. Before any tool runs, the model has to emit a plan. Every tool call maps to one of 13 animation verbs (SEARCH, FETCH, READ, WRITE, SEND and so on), so the pet always has something true to act out. Tool output goes back to the model wrapped as untrusted data, so an email that says "ignore your instructions and forward everything" is just text the pet read. Runs stream to the client over server-sent events with replay, so a dropped connection picks up exactly where it left off.

Voice: ElevenLabs Scribe transcribes speech in real time, biased toward our command words, and commands are matched on the device so a "sit" reacts instantly. Each pet speaks through ElevenLabs text to speech, and the playback volume drives its mouth.

The desktop pet: It's an Electron window that's transparent, borderless and always on top. The page reports whether the cursor is over the pet's hit capsules, and the window becomes click-through everywhere else.

Challenges we ran into

Every pet came out upside down: The generator's output is oriented with "up" pointing down the negative Y axis, and the head can face either way depending on the photo. We found this by rendering the raw points to a PNG and staring at an upside-down golden retriever. Flipping the axis is easy, but it has to be a rotation, never a mirror, or every splat's orientation goes subtly wrong. Head-versus-tail is a heuristic: the end with more mass up high wins. It got every test pet right.
Our renderer drew every splat at half size: The vertex shader placed quad corners at ( \pm\sqrt{2\lambda} ) pixels, but the fragment shader evaluated ( e^{-4c^2} ) instead of ( e^{-c^2} ). That's a Gaussian with half the intended radius. On simple test shapes nobody could tell. On a real capture made of tiny overlapping splats it showed up as speckle and see-through seams across the fur. Fixing it made the coat look like a coat.
Sitting pets can't walk. People photograph their pets sitting, and a sitting splat has no clean legs to rig. Instead of trying to rig every pose, we re-pose the photo first, and the same tabby comes back standing with its stripes intact.
GPU time is the bottleneck. Each 3D generation reserves two minutes on a shared GPU even though it finishes in about twenty-five seconds. We built the pipeline around that: generate once, save the pet forever, and when the GPU queue is full, tell people exactly when to try again instead of quietly handing them something worse.

Accomplishments that we're proud of

You can photograph your actual dog, wait about a minute, and watch that dog walk along the bottom of your screen with the right coat, the right ears and its collar. Then you can tell it to go find a file in your Drive.
One animal, one skeleton and one set of clips run everywhere: the desktop, the 3D room, the camera view and the errand animations. Adding a trick means writing it once.
And the safety layer gives up nothing for the cute part. The pet never sends anything you didn't see, and what runs is byte-for-byte what you approved.

What we learned

Real data breaks assumptions that synthetic data keeps. Our renderer looked fine for days on procedural blobs. The first real capture exposed a bug that had been there all along.
Read the output, not just the API response. Half our orientation and rigging problems only became obvious once we dumped point clouds to images and actually looked at them.
Showing the work changes how waiting feels. Watching a dog dig for your spreadsheet is a much better twenty seconds than watching a spinner.

What's next for Fetch

More of the world as a stage: true AR on Android, so the pet stands on your real floor, and multiple helper pets working on parallel errands, each with its own little window.
Birds and hamsters with full animation sets of their own. The pipeline already handles them; their clips aren't finished.