RoboResponder

System Diagram

Inspiration

During an emergency, navigating an unfamiliar building can be difficult, especially for people with visual, mobility, or other impairments. We wanted to build an autonomous robot that could understand its environment through both vision and voice to guide people toward safety. This led us to RoboResponder, an emergency navigation rover designed to identify exit routes, structural cover, and hazards while keeping evacuees informed out loud.

What it does

RoboResponder acts as an interactive emergency guide that pairs real-time voice triage with autonomous spatial navigation. When activated, the robot asks what emergency has occurred and listens to evacuees report hazards such as fires, earthquakes, or tornadoes. After confirming the emergency, it vocalizes clear departure instructions like "Attention everyone, follow me!" and begins navigating. While moving, the robot uses camera vision to spot emergency exit doors, sturdy tables for cover, or open corridors, and executes driving commands to guide people along a safe path. Every few movements, the robot provides spoken updates through a speaker to keep evacuees calm and informed.

How we built it

We used an XRP robot equipped with dual DC drive motors, quadrature encoders, and a SparkFun RP2350 microcontroller as our physical platform. A webcam captures real-time video frames, which are processed with OpenCV to supply visual input to Gemini Robotics ER 2. Gemini Robotics ER 2 evaluates scene geometry, maintains a sliding visual memory across recent frames, and directly triggers driving tools like moving forward, steering, and turning through automatic function calling. Alongside spatial vision, we integrated Gemini 3.8 Live to manage conversational voice interaction, handling microphone audio triage and streaming low-latency synthesized speech through an external speaker. A central Python coordinator links these models to the robot over USB serial, communicating with custom MicroPython firmware that provides closed-loop motor speed control, encoder telemetry, and a 1.5-second safety watchdog.

Challenges we ran into

Our biggest challenge was integrating vision, voice AI, and embedded hardware into a single responsive pipeline. We had to connect the live webcam stream, OpenCV, Gemini's streaming capabilities, and the XRP hardware without introducing disruptive latency. We also had to fine-tune closed-loop motor speeds and action durations so AI tool calls resulted in smooth physical travel on the floor. In addition, we had to manage multimodal visual context across sequential frames to avoid exceeding API token limits while ensuring voice announcements finished before the wheels began turning.

Accomplishments that we're proud of

We successfully combined Gemini Robotics ER 2 and Gemini 3.8 Live into a working dual-agent robotics system. We were able to have Gemini ER 2's spatial reasoning directly control physical rover motors in response to live camera feeds, while simultaneously enabling evacuees to verbally report hazards and hear spoken guidance. We also developed a real-time OpenCV HUD with a stage reset switch, allowing us to safely halt and reset the robot during live demonstrations.

What we learned

We learned how to connect multimodal embodied AI models with physical robotics and embedded microcontrollers. We gained hands-on experience with real-time bidirectional audio streaming in Gemini 3.8 Live, microphone capture in room environments, and managing visual tokens across sequential camera frames. We also learned how to implement closed-loop motor control, real-time serial telemetry, and safety watchdogs in MicroPython to ensure reliable hardware behavior.

What's next for RoboResponder

Our next step is to mount an onboard computer, such as a Raspberry Pi or NVIDIA Jetson, directly onto the chassis so the robot can run completely untethered from a host laptop. With onboard compute, we want to deploy a local LLM or Vision-Language-Action (VLA) model directly on the device, allowing the robot to navigate and triage emergencies even when internet connectivity or building power is knocked out during a disaster. We also plan to scale up the physical size of the robot, which will give the camera a higher sightline over obstacles and give the rover a larger presence that commands more attention from evacuees. Finally, we want to focus on improving obstacle avoidance and implementing multi-room mapping so RoboResponder can reliably guide people through larger, crowded building layouts.