Sophi - The Physical Augmentation Walker

Sophi is an AI-powered walker with an always-on voice agent for Q&A and GPS navigation, plus a linear lift with robotic arms. Built-in features include collision avoidance and fall detection.

Comment 2

Sophi — The Physical Augmentation Walker

Inspiration

Last year, one of our team members’ grandfathers suffered a serious fall on his way home and spent several months in the hospital. Seeing how a single fall could threaten his life, and how the resulting mobility challenges affected his independence, made us realize how much everyday freedom depends on being able to move safely.

We built Sophi for people like him.

Sophi is an AI-powered physical augmentation walker designed to help people navigate their surroundings, interact with everyday objects, and stay connected with the people who care about them.

Our vision also extends beyond elder adults. It includes the 340 million people who are blind or have low vision, whom our navigation technology can help explore the world with greater confidence. It includes the 1.7 billion people with mobility restrictions, who can use our robotic arms to open doors or pick up dropped objects. And it includes the billions of people who care about an elderly or disabled family member, giving them a way to check in on their loved ones through our companion app.

What it does

Sophi combines a walker with a conversational AI assistant, robotic arms, and navigation technology.

With a button on the handle, users can speak to a cloud-based agent through a microphone and speaker. The agent can answer questions, display information on a touchscreen, start GPS navigation, and operate Sophi’s SO-101 robotic arms and linear lift. The arms are intended to help with everyday tasks such as opening doors and picking up dropped objects.

We use Anthropic’s Claude for the assistant's reasoning and tool-use loop. It can decide when a question needs a fresh distance reading, location, weather, or a screen card, then use the result to answer. We use Cartesia for speech recognition and for Sophi's voice.

Distance sensors detect nearby obstacles and trigger haptic feedback to guide the user. Practical additions, including a bike light and bottle holder, support everyday use.

Our companion app lets caregivers view the user’s location, check in through Sophi’s camera, and receive messages sent from the walker. To make sure join requests, approvals, membership, and messages can update live, we use SpacetimeDB for the care circle.

We also built an AI-guided customization experience that learns about a prospective user’s needs and recommends which modules their walker should include.

How we built it

Sophi runs on a Raspberry Pi 5 connected to three ESP32 modules. This hardware connects the walker’s sensors, controls, and physical components with the cloud-based agent.

We integrated the touchscreen, GPS, microphone, speaker, distance sensors, haptic feedback, robotic arms, and linear lift into one physical platform. To assemble the prototype, we soldered four circuit boards and designed and 3D-printed more than 30 custom parts.

The project brought together mechanical design, electronics, embedded programming, AI, and app development. Every feature needed both a physical implementation and a way for the user to access it.

Challenges we faced

The central challenge was integrating so many different systems into a single walker. A spoken request needs to travel from the user to the agent and then become an action on the appropriate hardware.

Physical integration was another challenge. The arms, lift, sensors, screen, and wiring all had to fit around a walker that someone would hold and move with. Custom-designed parts helped us adapt the components to that shared structure.

We also had to consider different accessibility needs. Voice interaction, visual information, and haptic feedback each serve a purpose, and deciding how they should work together shaped the design.

Finally, the project required a ton of electrical debugging, including shorted solder joints, overloaded USB buses, and poor connections.

What we learned

Building Sophi taught us to think about the whole interaction, such as what someone wants to do, how they communicate that intent, and how the device responds in the physical world.

We also learned how closely hardware and software decisions depend on one another. Component placement affects usability, mechanical design affects what the software can accomplish, and the interface determines whether those capabilities are accessible.

What’s next

We want to turn Sophi into a real product. Our next priorities are refining the prototype, testing reliability and usability, learning directly from potential users and caregivers, and taking it to market.

We have already connected with a major prosthetics nonprofit to discuss Sophi’s future. As we continue developing it, our motivation remains the same: to help people who need it most move through daily life with greater safety and independence.

Additional Work

To control the robot arms, we developed a custom GPT-controlled harness, tested control with the Dream Labs API, and evaluated some open-source AI models for the SO101 arms. The GPT-controlled harness worked, but was far too slow. The Dream Labs API controlled the arms, but did not seem to behave reliably out of the box. None of the open-source models tested were reliable either. In the end, we developed our own harness which uses IPhones to control the SO101 robot arms.

Built With

Share this project:

Updates

Submission history