Goosetriever

Inspiration

Imagine playing with your beloved goose plush and accidentally dropping it. Usually, you would just bend down and pick it up.

But for someone with limited mobility, that small accident can mean waiting for help. A dropped phone, inhaler, or pair of glasses can interrupt their independence.

Thinking about our own elderly family members, we wanted to explore a robot that could help with this everyday task. Our first retrieval target was a goose plush—a fitting companion for a project built at the University of Waterloo.

Meet Goosetriever.

What it does

Goosetriever is an assistive robotics prototype built around a simple request:

“Goosetriever, can you help me pick up my goose plushie?”

It combines spoken requests and camera images through a multimodal AI model, locates the target, and uses a lidar map to navigate toward it. A robotic arm with a learned grasping policy is designed to complete the pickup and handoff.

Our browser dashboard makes the robot’s movement visible: it shows the room map, obstacles, the robot’s position, its movement history, and its home location. We can click a destination, draw a path, send it home, or take over with manual controls.

We demonstrated autonomous navigation and exercised voice-guided approaches to the goose. The arm’s grasping system and handoff were developed separately; later integration runs reached the object but encountered a gripper-servo fault. Making the entire pickup-and-return sequence reliable remains our next milestone.

Goosetriever won the Huawei OMNI Live Challenge sponsor track at Hack the North 2026.

How we built it

Multimodal interaction. Through Huawei’s OMNI Live challenge, we integrated Qwen-Omni using a cloud API. A request includes the user’s audio and a frame from the robot’s camera. The model returns a spoken response and a structured intent that the application can use. This connects what the user says with what the robot sees.

Navigation and control. A Raspberry Pi controls four DDSM115 wheel motors over RS485 and reads an RPLIDAR A2M12 and an MPU-series gyro. The laptop builds a log-odds occupancy grid, adds clearance around obstacles, plans routes with A*, and follows them using pure pursuit. The gyro supplies heading information because wheel slip makes turning estimates from the encoders unreliable.

Vision. An Intel RealSense D435i supplies camera frames through a Raspberry Pi running QNX. YOLO-World runs on the laptop to locate the goose. Using the camera’s mounting geometry and the assumption that the object rests on the floor, we convert its image position into a target relative to the robot.

Manipulation. An SO-101 arm on an RDK S100 runs an ACT—Action Chunking Transformer—policy trained on teleoperated demonstrations of picking up the goose. Training moved to Hugging Face after problems with our Baseten run. Inference runs locally on the arm board, with a separate handoff pose for presenting the object.

Stop mechanisms. The drivetrain includes a link watchdog, motion deadman, e-stop latch, and lidar safety checks. We also built and bench-tested a QNX person-detection supervisor that produces a heartbeat signal only while conditions are safe. Its physical connection to the drivetrain remained unfinished.

The final integration setup used the laptop for audio, detection, and mission sequencing, the QNX Pi for camera capture, the navigation Pi for the drivetrain, and the S100 for the arm.

Some software foundations predated the event. Hardware integration, navigation development, and debugging continued throughout the hackathon, with further integration experiments after submission.

Challenges we ran into

Getting every subsystem to agree. Voice, vision, navigation, and manipulation needed consistent interfaces, coordinate frames, and sensor-age information.
Wheel slip and sensor alignment. Incorrect motor directions, gyro bias, and lidar orientation produced behavior that initially looked like navigation bugs.
Power and connectivity. Brownouts, USB failures, and network stalls repeatedly interrupted integration.
Recognizing a fallen goose. A plush lying on its side looked very different from an upright goose. Broader visual prompts helped.
Arm reliability. Camera disconnects, servo communication failures, and a missing gripper-servo response prevented reliable full-loop pickup.
Training infrastructure. We changed training providers while also collecting demonstrations and debugging the hardware.

Accomplishments that we're proud of

Winning the Huawei OMNI Live Challenge at Hack the North 2026.
Getting a physical robot to map its surroundings, follow routes, and return toward a saved home position.
Combining the user’s speech and the robot’s camera image in a multimodal interaction.
Building a dashboard that made navigation understandable and helped us diagnose failures.
Training a grasping policy for our goose plush and running inference on the arm board.
Bench-testing a QNX supervisor that stopped its heartbeat when its detector failed.

What we learned

The biggest lesson was to integrate earlier. A working camera, a working arm, and a working navigation system still need careful testing together.

We learned to measure the robot we actually built: wheel direction, turning slip, camera placement, network delays, and power behavior all mattered.

We also learned to distinguish a component test from a complete robot demonstration. Simulation helped us develop quickly, but physical hardware exposed problems that software tests alone could not.

And next time, we are bringing more tools, spare cables, and a multimeter.

What's next for Goosetriever

Resolve the gripper fault and demonstrate repeatable pickup-and-return runs.
Connect the QNX supervisor to the drivetrain and extend coordinated stopping to the arm.
Package the working voice, camera, and mission components into a reproducible startup flow.
Improve close-range perception and integrate depth-camera obstacles into live navigation.
Expand beyond the goose with additional demonstrations and evaluation.
Refine the chassis, power distribution, and cable management.
Explore accessibility requirements with potential users and caregivers.