700-SF11 Vamos

Emcee for events and spaces; blend human and AI.

Comment

Inspiration

We want to blend our love of telepresence / XR-assisted experiences, social robots, and event spaces to help people connect more easily. We’re excited about turning “awkward mingling” into playful, guided social encounters and building community through robots as social catalysts.

What it does

The aim is to create an XR-human-powered and LLM-souled robot duo (Timon and Pumbaa, if you will – but we call them Reachy and Reach :) ) that acts as a host and guide in spaces where people gather and expect to connect.

Reachy is the side-kick with its unique expressions, and Reach is the wheels so that Reachy can reach people! Reachy is the mischief-multiplier; Reach is the party host who knows everyone and everyone’s business, haha.

Our bots:

  • break the ice with small talk and playful prompts
  • help guests meet each other and start conversations
  • act as a navigation guide in the space
  • serve as a general conduit for human-to-human connection

Teleoperators in XR can steer the vibe as “neutral” or “funny,” while the robots handle the flow of the interaction.

How we built it

We’re running LLMs in Docker and connecting them via an Android SDK to:

  • a temi robot for semi-autonomous navigation, audio, and video telepresence, running our custom Android application
  • a Reachy Mini companion robot via its SDK for expressive, interactive behaviors

On top of that, we built a Unity app for Meta Quest that lets the human “behind” the robots:

  • communicate with the LLM-driven bots
  • guide their personality and tone (neutral vs. funny)
  • influence their prompting and motion toward people as they navigate the space

Our Android app (Temi), Unity app (Meta Quest 3), and a Python app (Reachy Mini on Win) all talk to each other over the Agora SDK with sub–200 ms delay!

Challenges we ran into

  • Managing reliable connections between the dockerized app and the robots while everyone (and everything) lived in different physical locations and networks.
  • Designing for our ideal scenario where all team members can seamlessly communicate across XR, robots, and backend services without the wiring getting in the way of the experience. Each build takes at least 15 minutes, so after ~40 builds you’ve already lost a day that you don’t get to spend actually testing and running what you’re building.

Accomplishments that we're proud of

  • Setting up our LLM-driven, human-in-the-loop interaction as the core SOUL of the system
  • Wiring up the robots, XR, and remote access as the SPINAL CORD that carries intent and expression into the physical world
  • Getting multiple teleoperators co-piloting multiple robots toward a shared social goal (hosting a party!)

What we learned

  • We absolutely need another day 😅
  • Coordinating XR, robots, LLMs, and networks is hard—but incredibly fun and full of surprising design questions about social interaction, consent, and comfort.
  • Crazy task pipelining, using the build times as redesign and revisioning times.

What's next for 700-SF11 Vamos

  • Fund us for our bot!
  • Turn this prototype into a deployable “robot host” service for events, meetups, and public spaces -- — starting with the next SensAI hackathon in SF!!
  • Collect more interaction data to improve the social behaviors of the robot duo
  • Expand beyond “neutral” and “funny” into richer emotional palettes and roles (hype host, gentle guide, calm listener, etc.) with more gestural expressions with Reachy.
Share this project:

Updates

Submission history