Inspiration

We wanted to explore a simple question like what if your virtual reality headset could give you more than games and entertainment? What if it could give you a companion? What if it has agentic actions?

Pets bring joy, comfort and a sense of connection to our everyday lives. But not everyone can own a real pet due to space, time, allergies, or other circumstances.

That inspired us to build WithMe which is an interactive virtual pet experience that brings the warmth and personality of having a dog into your virtual space.

Instead of creating another game character that simply follows predefined commands, we wanted to make a companion that feels alive, has its own behaviours, responds to interactions, and makes the virtual environment feel more personal.

What it does

WithMe brings an adorable Shiba Inu companion a.k.a Mochi into an immersive virtual environment.

The dog has its own behavioural system and can independently explore, play, rest, sleep, eat, and seek attention and users can interact with their companion through actions such as petting, playing fetch, feeding, and calling it over.

  • Social interaction: The dog greets users, approaches them, and seeks attention.
  • Play: It interacts with a ball, plays independently, and invites users to play fetch.
  • Needs and behaviours: Hunger, energy, and sleepiness influence its behaviour.
  • Companion routines: It explores its surroundings, visits its bed, eats from its bowl, and sleeps when tired.
  • Responsive animations and sounds: Animations and audio cues make interactions feel more expressive and engaging.

Our goal is to make the dog feel like a companion with its own personality rather than a character waiting for instructions.

How we built it

  • Meta IWSDK: Spatial application framework and immersive experience development.
  • Three.js: 3D rendering and virtual environment.
  • TypeScript: Application logic, behavioural systems, and interaction management.
  • WebXR: Immersive VR capabilities and spatial interactions.
  • IWER: Desktop WebXR emulation and development testing.
  • 3D character animations: An animated Shiba Inu model with different movement and behavioural states.
  • Audio effects: Dog vocalisations and environmental interaction sounds to add personality.
  • Supabase: For Backend agentic actions

Challenges we ran into

A dog is not simply a 3D model that moves from one coordinate to another. Its movements need to communicate intent through body language, head orientation, anticipation, weight shifts, and natural transitions between animations.

Another challenge was coordinating autonomous behaviours with user interactions. We needed to ensure that the dog could independently perform activities while still responding appropriately when the user wanted to interact with it.

We also faced challenges with WebXR development and testing, particularly around desktop emulation, controller interactions, and capturing immersive experiences on standalone hardware.

Accomplishments that we're proud of

  • Creating an immersive virtual environment for a Shiba Inu companion.
  • Implementing 13 distinct behavioural actions.
  • Building an autonomous behaviour system influenced by the dog's virtual needs.
  • Enabling interactions around playing, feeding, petting, and companionship.
  • Integrating animated character behaviour with contextual audio effects.
  • Exploring how spatial experiences can be developed using web technologies and Meta IWSDK.

What we learned

The real challenge is creating the illusion of life through behavioural design, contextual responses, animation timing and small interactions.

What's next for WithMe?

  • Voice interaction: Allow users to call their pet and give natural voice commands.
  • More expressive behaviour: Improve animation blending, body language, emotional responses, and personality.
  • Personalisation: Allow users to customise their pet's appearance, name, personality, and environment.
  • Smarter AI behaviour: Explore AI-driven decision-making to make the companion more adaptive and less predictable.
  • Expanded environments: Bring the companion into more virtual spaces and everyday experiences.

Built With

Share this project:

Updates

Submission history