Bracket bot grasping wheelchair
Head camera POV
Simulated environment map

Inspiration

The idea for this project came from one of our team members, whose grandparent experienced a stroke and now lives with aphasia. Seeing how difficult it can be to communicate and complete everyday tasks made us think about how intelligent systems could provide more support at home, especially for those that do not have constant access to someone's assistance. Wheelchair users often have to rely on others, causing proneness to accidents, dropping something important, etc. We want to create a solution that not only could give family members peace of mind when they cannot be there to assist, but also provides autonomy and independence for the user.

What it does

Remy is a simulated wheelchair-assistance robot that helps a user move through an indoor space. It approaches a wheelchair, positions its grippers on the push handles, and moves the chair toward a requested destination. The user can give voice commands to direct the robot, while its autonomous navigation uses camera images and ArUco markers to recognize rooms and find a route. For tasks that need a human’s guidance, the user can switch to Meta Quest 2 VR controls. The headset shows the robot’s camera view, and the controllers let the user steer the robot and move its grippers.

How we built it

We built a wheelchair-assistance prototype in MuJoCo. We imported a robot model, added a wheelchair and an apartment environment, and connected destination commands to autonomous navigation and added voice commands so a user can direct the robot by naming where they want to go.

For manual control, we integrated a Meta Quest 2. Its controllers set the robot’s gripper targets, while a live view from the robot’s head camera appears in the headset. For navigation, we used camera images, wheel odometry, and ArUco signs near doorways to help the robot identify rooms and build a map.

Challenges we ran into

Finding the wheelchair and identifying a reliable approach position took more than simply moving toward it: the robot had to line up behind the chair without blocking its own access to the handles. Grasping was another challenge because small errors in gripper position could cause the hands to miss or slip off the handles. We worked on arm positioning and checked contact at the grasp points in simulation. Wall collisions were also difficult to simulate convincingly. We had to refine collision geometry and clearance checks so the robot and wheelchair would respond to barriers while still allowing the grippers to reach the handles.

Accomplishments that we're proud of

We integrated VR controls, voice commands, and autonomous navigation into one prototype. The Quest controllers let a person guide the robot’s hands from the robot’s camera perspective, while destination commands let the robot navigate on its own.

What we learned

MuJoCo taught us how much a useful robotics simulation depends on accurate joints, contacts, and collision geometry. We learned to use ArUco tags as recognizable landmarks for room identification and localization. We used OpenCV to generate and detect those tags in camera images, estimate their position, and process visual information for navigation.

What's next for Remy

Next, we want to use VR to interact with more objects in the environment. We also want to explore autonomous tasks that could help with daily life, including assistance with getting on and off a wheelchair and tasks in the bathroom. These are future goals beyond the current simulation prototype.