Goosetriver

Inspiration

Imagine that you are playing with your beloved goose plush, and you drop it by accident. Not a problem, just bend down and grab it.

But what if you are 80+ years old, or if you suffered from a spinal cord injury? Odds are you are in a wheelchair, and bending down to pick things up on the ground is significantly more difficult and dangerous. Your goose plush remains confined to the ground, away from your warm embrace.

While this sounds like a pretty silly premise, this is actually a problem a lot of elderly and those with conditions like ALS, cerebral palsy, and general spinal cord injuries deal with very frequently. Published research on assistive robotics found it happens an average of 5.5 times a day, and when it happens, retrieval can take up to two hours if unassisted.

Seeing that our elderly family members are at risk of suffering from this, we wanted to build a practical, deployable solution that was able to pick up any dropped objects and return them to the user with ease.

What it does

A fully voice-controlled, autonomous fetching robot designed to assist seniors and those with spinal cord injuries. Our solution is powered by Huawei's OMNI Live API for voice communication and task orchestration, alongside edge-deployed vision and navigation inference using QNX and RDK. Once the user speaks a command like:

"Goosetriever, can you help pick up my goose plushie"

It navigates autonomously towards the object while actively maneuvering around obstacles using a depth camera and lidar. Upon reaching the item it uses its arm to grasp it, and returns to the user.

By eliminating the hazardous strain of bending over, it empowers individuals to reclaim their daily autonomy and independence, making everyday pickups as effortless as they used to be.

How we built it

Goosetriever splits into a cloud brain, a central Pi that talks and sees, and two edge boards that move.

The cloud decides what, never how. An LLM on the OpenAI API picks which skill runs next — goto, approach, pick, deliver, ask_user, refuse — and the registered tool list is the entire action space. The model cannot emit a velocity. It chooses a destination; a lidar map, a distance transform and A* choose the path there.

Central Pi, running QNX 8.0. Voice, eyes, and a safety interlock on one chip. Huawei's OMNI Live API carries the whole exchange: one call takes the camera frame, the raw audio and the language together, so "bring me the goose — I think it's by the couch" can be grounded against what the camera actually sees. The same board runs a TFLite person detector from oss.qnx.com at SCHED_FIFO 50, holding a 20 Hz heartbeat on a GPIO line into the drivetrain's e-stop. Running a network-facing coordinator beside a hard-real-time safety loop is only safe on a microkernel — we measured 20–30 µs of scheduling lateness on a 25 ms tick with the detector saturating the CPU, and 38.5 ms per inference on-device.

Nav Pi, the spine. Four DDSM115 hub motors over RS485 — the driver and its bench tool were written with Codex, which measured what the datasheet could not tell us: which motor ID sits on which side, which way is forward, and how many encoder counts a wheel turn is worth. Alongside them an RPLIDAR A2M12 and an MPU-6050 gyro, plus every watchdog that stops the robot: a 300 ms link watchdog, a 500 ms motion deadman, an e-stop latch, and a lidar safety bubble. Heading comes from the gyro because a skid-steer's tyres lie — ours claims 1.77× the rotation it actually produces. Zero-velocity updates took drift from 7.5 °/s to 0.000° over 10 seconds.

Navigation. We combine 2D lidar with a forward-facing depth camera to map the room in real time. Every lidar revolution goes into a log-odds occupancy grid at the pose it was taken from. A Euclidean distance transform (3 ms, hand-written in NumPy) inflates obstacles by the robot's turning circle, then A* (1 ms) plans through it and pure pursuit drives it — guiding the robot from the user to the object and back with no manual intervention.

RDK-S100, the hand. A custom ACT (Action Chunking Transformer) policy trained from scratch on teleoperated demonstrations, running inference locally on the board. Vision, orientation detection and the gripping sequence for the goose-like neck all execute on local edge compute. A grasp can't wait for a network round trip; ACT emits 100-action chunks so it doesn't have to.

Challenges we ran into

Integrating all the different systems and layers — voice commands to understand the user, depth camera and vision to detect the object, navigation to plan its way there, and the arm to grasp the goose
Arm and camera disconnects, and packet loss while recording episodes for the ACT policy
Cloud training for the ACT policy crashing on Baseten, so we pivoted to Hugging Face cloud training
Attaching the arm and the chassis in a stable manner
General hardware and power issues
Cable management
Installing all the different packages and dependencies across boards

Accomplishments that we're proud of

Integrating a full autonomous robot with navigation and obstacle avoidance, in real time and on-edge
A safety layer that survives its own failure: kill the detector, unplug the QNX board or cut the wire and the drivetrain stops. A steady line means stop, so a crash can't quietly read as "safe." Frame capture to stopped: 81–94 ms
Experimenting with and combining many different kinds of devices
Training a policy from scratch to pick up the goose

What we learned

How to integrate the sponsors' APIs and other technologies
Bring more tools to the venue — the hardware lab is very much not guaranteed to have what you need
Start integrating everything earlier
Move inference onto the NPU. We had very little time, so we didn't risk setting up the BPU for the ACT policy