Inspiration
There are many that we know, as a team, that have substantial disability when completing self-supportive tasks. Complaints of tasks often taken for granted – drinking water or taking necessary medication – become the norm and you wish for solutions that don’t require constant monitoring and assistance. We thought of some vague robotic solutions, but realized the issues regarding cost of implementation are a real barrier for most. We decided on taking advantage of cost-effective solutions that utilized the strength behind the eyes and voice. Our emphasis on the pair of stimuli is done so as many victims of age or preoccupation have either one often at their disposal. Since coming to these conclusions, we believed that being able to synthesize the two would be key in bringing together a worthwhile product.
What it does
Our robot gives quadriplegics back some of their autonomy by empowering them to “grab” objects for themselves again.
How we built it
We brainstormed approaches to how our synthesis of cost-effective materials and decided printing the LeRobot pieces and buying the respective actuators and boards would be best for our purposes. As for the eye tracking hardware, we brought along multiple pi cams and were able to jerryrig/solder a set up on frames to synthesize the various perspectives we had in our set up.
Challenges we ran into
Campus Wi-Fi: Device-to-device connectivity restrictions led us to use an Archer T3U and a local 2.4 GHz hotspot.
Limited USB-C cables: We soldered power leads so the outward-facing camera shared the eye-tracker assembly’s supply while streaming video wirelessly.
Shared compute: The A2000 was often occupied, prompting experiments with local inference and Modal cloud GPUs.
SmolVLA integration: Predictions worked, but camera mappings, joint conventions, calibration and normalization remained obstacles to reliable grasping.
Recording and cameras: Recording timeouts discarded takes, and competing processes attempted to access the same USB cameras.
Constrained vision: Small objects were difficult to recognize in the low-resolution grayscale glasses feed, motivating simpler image processing and information from the workspace camera.
Accomplishments that we're proud of
We assembled and calibrated the SO-101 arm, built the eye-tracking glasses, and got them working together so a person could select an object with their eyes and have the arm pick it up. Some of the work we're proudest of came from fixing problems at the venue. When campus networking blocked camera communication, we set up a local hotspot. With too few USB-C cables available, we soldered a shared power connection for the cameras. Sharing the A2000 also pushed us to try model inference both locally and in the cloud.
What we learned
Our SmolVLA experiments taught us how much work sits between getting a model to produce predictions and getting a robot to act on them. Camera inputs, joint calibration and recorded demonstrations all had to match the arm we had built. We ended up using OpenCV for eye tracking and geometry, YOLOE for object recognition, and inverse kinematics with taught grasp settings for the demo. That made it easier to inspect each step when something went wrong. The interface needed care too. Looking at an object shouldn't automatically mean picking it up, which is why we used a deliberate blink to confirm. Next, we want feedback from people with limited hand mobility about which tasks they would actually want Look to help with.
Log in or sign up for Devpost to join the conversation.