Inspiration

Imagine Margaret, a 78-year-old recovering from a fall. After her daughter leaves for work, her water and phone are across the room, but she cannot safely get up to reach them.

We built ENGRAM Assist for everyday moments like this. We wanted to give people with mobility challenges more independence, with voice and gesture controls that offer different ways to ask for help.

What it does

ENGRAM Assist is a wheeled assistance rover with an arm and gripper that users guide to fetch lightweight objects around the home.

Users can give instructions such as “move forward four times” and “pick it up,” or use hand gestures to drive, grab, release, and adjust the arm. Movement is guided step by step. A dashboard displays the user camera and robot’s view alongside gesture indicators, speech transcripts, and command queues, helping users understand what the rover heard and why it moved.

The rover accepts only supported commands and rejects unclear instructions.

How we built it

We built the rover using a Hiwonder MasterPi and Raspberry Pi 5, with a robotic arm, gripper, front ultrasonic sensor, and a Logitech C920 webcam for audio and an additional camera view.

We used Lovable to build our React and TypeScript frontend, integrating MediaPipe Hands for gesture tracking and Three.js/React Three Fiber for an interactive rover model. Whisper transcribes speech into supported actions, while a Python/FastAPI backend sends HMAC-signed commands to the robot.

The system communicates over a local hotspot, with the robot camera streamed through an SSH tunnel. The ultrasonic sensor supports obstacle checks and a slow approach during pickup.

Challenges we ran into

Connecting to the Raspberry Pi over SSH and getting it onto our network took time, especially without a monitor for troubleshooting. Linking hand gestures to robot movements was another challenge: controls that initially worked stopped responding, requiring repeated debugging and testing.

We also compressed webcam video to keep it running alongside microphone input, resolved mismatched command-signing secrets, and repeatedly adjusted the arm’s pickup position against the real floor. Getting each component working was only part of the challenge; making them work reliably together took the most effort.

Accomplishments that we're proud of

We brought voice control, browser-based gesture tracking, camera feeds, and physical robot control together in one night. Most rewarding was seeing a spoken instruction make the real rover move, grab, and carry an object. Our integrated code also passed 325 automated tests, alongside hands-on hardware testing.

What we learned

Working hardware takes more than working code. Network connections, USB bandwidth, and small mechanical adjustments can determine whether a demo succeeds.

We also learned that an assistance robot needs predictable behavior and clear feedback. Users should know what it understood and remain in control of what happens next.

What's next for ENGRAM Assist

We want to work with people with mobility challenges, caregivers, and occupational therapists to identify useful tasks and guide supervised testing. Next steps include a rear sensor, spoken feedback, and more adaptable gesture controls.

We also plan to evaluate recognition across accents, skin tones, lighting conditions, and ranges of movement so the system can support more people reliably.