Inspiration

Remote-controlled robots assume two free hands and a controller. That leaves out people with limited hand mobility, and workers whose hands are busy, gloved, or holding tools. A fixed camera can't help either: it sees one angle and can't go look. We wanted a rover that anyone can steer by looking and speaking.

What it does

Lookout Rover is a browser-controlled XRP rover. The operator turns their head to steer and says short commands ("go forward," "reverse," "turn left," "turn right," "stop," "resume," "recenter") to move. A phone mounted on the rover streams a live view of its surroundings.

  • Head steering: the laptop webcam tracks head rotation, and the rover turns to match. The rover closes the loop with its own IMU heading.
  • Voice commands: the screen shows what the mic heard and whether the rover accepted it, so the operator always knows what happened.
  • Layered, local safety: the rover stops on Stop/Esc, a hidden tab, a lost face, stale telemetry, an obstacle under 20 cm, a missing front echo, motor power loss, or stalled wheels. Faults latch until fresh data arrives and the operator explicitly resumes.
  • Hoot the owl: our mascot keeps watch and reacts to the rover's state and telemetry. Hoot only reads what the rover reports and never drives the motors.

How we built it

  • Rover: an XRP vehicle with a Red Board running MicroPython and an XRPLib controller (differential drive, IMU heading, front ultrasonic range, wheel encoders, motor power reporting).
  • Control app: a browser app hosted on GitHub Pages. It talks to the board over Bluetooth (Nordic UART) or USB serial using a small text protocol (STOP, ZERO, AIM angle base rate, TELEMETRY).
  • Head tracking: MediaPipe Face Landmarker runs in a worker. After seven stable samples set a neutral pose, head yaw becomes a bounded heading goal:

$\theta_{\text{target}} = \operatorname{clamp}\big(\operatorname{filter}(\theta_{\text{yaw}} - \theta_{\text{neutral}}),\ \pm 55^\circ\big)$

  • Voice: the Web Speech API maps speech to a small command vocabulary.
  • Phone view: the mounted phone's rear camera streams to the laptop over WebRTC (VDO.Ninja). This is separate from the motor-control link, so a video problem can't affect driving.
  • Diagnostics: an operator-visible panel shows frame, pose, write, and stop events for troubleshooting.

Challenges we ran into

  • Our Raspberry Pi setup wasn't working, so we changed plans and used a mounted phone for the rover's camera.
  • Making head steering steady: we calibrate a neutral pose, ignore tiny movements, filter the signal, and add rate assist.
  • Keeping safety local: the board has its own watchdog (400 ms without a motion update), so a browser or network problem can't leave the rover moving.
  • Running head tracking, the phone stream, and voice at once without blocking Stop.

Accomplishments that we're proud of

  • Several independent ways to stop the rover, with latched faults and explicit resume.
  • A control method that works with no hands at all.
  • A clear separation between steering, video, and safety, so each can fail without making the others unsafe.

What we learned

Safety logic belongs next to the hardware. A browser is a good control surface, but it shouldn't be the last line of defense. We also learned that "mic heard" and "rover accepted" are different things, and showing both makes voice control much easier to trust.

What's next

These are planned designs, not built features.

  • Visual navigation with OpenCV: read frames from the phone stream, detect and track objects, let the operator pick a target by click or voice, and approach it slowly with a stand-off stop. Manual steering and Stop always override.
  • VR headset or smart glasses: we don't have these devices this weekend, so the laptop webcam and mic stand in. The control logic doesn't depend on the input device.
  • Hazard inspection: the same human-in-the-loop control could check dangerous or inaccessible spaces from a distance.

Limits

This is an indoor prototype on flat floors. Head steering needs a steady pose, so it suits seated operators best. Voice commands are a fixed vocabulary, and the front ultrasonic sensor covers only a narrow area. There is no object detection or autonomous navigation yet.

Built with

XRP, Red Board, MicroPython, XRPLib, Web Bluetooth, Web Serial, MediaPipe, Web Speech API, WebRTC, VDO.Ninja, JavaScript, GitHub Pages

Built With

Share this project:

Updates

Submission history