Pawlay

Inspiration

Pawlay started from a simple observation: new dog owners often struggle more with timing and consistency than with motivation. They love their dogs, but remembering when to click, when to treat, and how to lure cleanly is hard in real time. With Meta Quest 3’s passthrough and on‑device AI, it became possible to “watch” the real dog and offload the marking and coaching to the headset. The goal was to build something that feels like a calm trainer standing next to you, not a game you disappear into while your dog waits.

What it does

Pawlay uses passthrough on Quest 3 and an on‑device computer‑vision model to recognize simple dog behaviors like sit, stand, and down. Once the dog is anchored, a 2D panel floats above them in the room showing three‑frame visual guides, current instructions, and a subtle streak of paw icons for progress. When the model detects that the dog has done the selected behavior, Pawlay plays a clicker sound, flashes a “CLICK” indicator on the panel, and prompts the owner to feed a treat. The app’s design is intentionally minimal and dog‑calm, using soft blue and yellow accents so the owner can focus on their dog, not the UI.

How we built it

The prototype is an Android Spatial SDK app running on Meta Quest 3 with passthrough enabled and hand tracking for micro‑gesture input. The dog‑pose pipeline runs on device, combining a keypoint model for dog joints with a small classifier that outputs sit/stand/down plus confidence scores. A spatially anchored panel is placed above the dog using bounding‑box and hand location, then updated every frame with smoothed state from the CV model. Interaction is handled with thumb‑tap microgestures on the user’s hand, which control onboarding, anchoring, and starting or ending drills, without big arm movements that might distract the dog.

Challenges we ran into

The biggest challenge was noisy, imperfect vision models. The dog detector can jitter, briefly misclassify, or even lock onto a couch if the scene isn’t clean. An example of how noise and even the pose of the trainer affects the model's output is shown at the end of the video. From the trainer's perspective, the model is able to cut through the noise to recognize the pose, but a slight change in orientation completely changes the output of the model. To keep the experience believable, Pawlay uses temporal smoothing, hysteresis, and conservative thresholds so only stable behaviors trigger a click. Another challenge was designing MR UI that feels friendly but not over‑stimulating for a real dog in the room; that meant cutting features like complex swipes and keeping color, motion, and sound extremely restrained.

Accomplishments that we're proud of

Pawlay anchors a live UI element directly above a real dog and keeps it stable enough for a demo, even with a noisy model and a moving animal. The app also manages to teach a new interaction pattern—hand microgestures—without pulling attention away from the dog. Most importantly, the core loop actually works in a short real‑world drill: ask for sit, watch the panel, hear the auto‑click, give the treat.

What we learned

Designing for “human + real dog + headset” is very different from a normal VR game. Small choices in animation, sound, and gesture complexity can directly affect the dog’s arousal and the owner’s confidence. The work reinforced how much UX and state logic can compensate for imperfect models: by smoothing, delaying, and softening feedback, the system feels more accurate than the raw predictions are.

What's next for Pawlay

Next steps include expanding the behavior set (e.g., stay, come, simple tricks), experimenting with longer training plans, and personalizing thresholds per dog over time. Pawlay could also integrate richer analytics and session history to help owners track progress. Longer‑term, the same approach could support professional trainers or remote coaching, but the immediate goal is to refine the core loop and ship a reliable, dog‑friendly mixed‑reality trainer on Quest.

Built With

• android
• kotlin
• tensorflow