Rollie

Inspiration

A walker provides physical support. We wanted to explore how it could also help someone understand their surroundings and decide where to go.

For someone navigating with limited vision, choosing a destination is only part of the journey. Understanding what is ahead, finding a person, and knowing what the device is doing all matter. We built Rollie around a simple question: What if a familiar mobility aid could see its surroundings and respond to the person using it?

Our answer combines a real motorized rollator, onboard perception, and an iPhone experience designed around spoken interaction, clear feedback, and deliberate user control.

What it does

Rollie connects a person's intent to a physical robot.

Through our iPhone app, users can interact by voice or touch, select a detected person, preview a destination route, inspect live sensor information, and access manual driving controls. The walker processes camera and LiDAR data onboard to inform local planning and motor commands.

Our working hardware demonstrations include person-following, manual forward and turning controls, and a live integrated sensor stack.

We also implemented destination selection, route previews, and navigation software using cached OpenStreetMap data and iPhone location. Completing an outdoor autonomous route remains a next step: our current location accuracy has not consistently met the system's navigation requirements.

Rollie is an assistive robotics prototype, with further physical validation needed before independent use as a mobility aid.

How we built it

We split the system across three computers, each with a specific responsibility.

The iPhone handles interaction. We built a native SwiftUI app with spoken guidance, on-device speech recognition on supported devices, destination selection, person-selection tools, and live robot status. A USB connection links the phone to the walker, and Core Location supplies positioning information.

The NVIDIA Jetson Orin Nano handles perception and planning. A Luxonis OAK-D S2 provides RGB images, stereo depth, and inertial measurements, while an STL27L LiDAR measures surrounding obstacles. PIDNet-S runs through TensorRT to classify surfaces such as sidewalks, roads, and terrain. Ground-plane estimation combines those labels with depth to identify raised and lowered regions. YOLO person detection runs on the OAK-D.

We fuse these inputs into a shared top-down representation used by the phone, local steering, and A* path planning. Keeping one perception pipeline gives different parts of the system a consistent view of the scene.

The ESP32 handles motor actuation. Two independently driven gearmotors motorize the rollator through custom mounts and a dual H-bridge. The Jetson sends structured commands over USB serial; the ESP32 generates PWM outputs and returns telemetry. Firmware starts disabled and includes a communication watchdog that disables propulsion when valid commands stop arriving.

Python and ROS 2 connect the robot software, while the app makes its state understandable to the user.

Challenges we ran into

The hardest problems appeared where software met physical hardware.

The walker saw itself as an obstacle. LiDAR returns from the frame blocked person-selection paths. We traced the issue to nearby scan points, adjusted the self-filter, and restricted the forward-only person planner to the region ahead. Person-following worked on the walker after these corrections.

Equal commands produced unequal control signals. Both motors received matching software requests, but PWM readbacks differed. We isolated the GPIOs, instrumented the firmware, and found repeated PWM resets in the control loop. Removing unnecessary rewrites produced matching readbacks and measured control voltages.

Starting everything at once reset the Jetson. The camera, LiDAR, controller, and charging phone created a power demand the initial supply could not support. We changed the supply and staggered sensor startup.

Connectivity needed more than a pairing screen. Bluetooth pairing and reconnection issues led us to add a wired USB connection. We also separated command receipt from confirmed robot state, so the app could report delayed acknowledgements and stale sensor data accurately.

Accomplishments that we're proud of

We brought mechanical hardware, embedded firmware, onboard AI, robot planning, and a native phone app into one physical prototype.

We are especially proud of getting person-following and manual control working on the actual walker, and of making the system inspectable through live sensor views, telemetry, and diagnostics.

Our perception runs onboard, and supported speech interaction runs on the phone. The central interaction does not depend on sending camera frames to a cloud AI service.

Just as importantly, we worked through failures with measurements: checking signals, reading logs, isolating components, and testing each correction on the system.

What we learned

A command being sent, a command being accepted, and a robot moving as intended are three different events.

That distinction shaped both our architecture and our interface. A connected phone does not guarantee fresh sensors. A spinning motor does not guarantee traction. A route on a screen does not prove the walker followed it.

We also learned that accessibility includes communicating uncertainty. Spoken interaction matters, but so does clearly explaining when the robot is waiting, disconnected, or unable to proceed.

What's next for Rollie

Our next milestone is a short, supervised outdoor route with reliable positioning. We also plan to calibrate encoder feedback and sensor alignment, measure stopping behavior and drivetrain performance, and physically validate depth-based hazard handling.

We want to test the interface with blind and visually impaired users so their feedback guides the next iteration.

The goal is to make Rollie easier to understand, easier to direct, and more dependable with every tested step.