Gaze2Grasp

Inspiration

Assistive robotic manipulation requires users to communicate their goals through interfaces that may provide far fewer controllable dimensions than the robot itself. Prior work on latent action representations addresses this mismatch by mapping low-dimensional inputs to coordinated robot motions. For users who can identify a desired object but cannot comfortably sustain manual control, a complementary challenge is to express that choice without repeatedly specifying the details of execution. In a tabletop task, a user may know which cup is needed while finding continuous steering or frequent control-mode changes difficult. This work considers users whose hand input is unavailable, difficult to sustain, or occupied by another activity, provided that a reliable channel for deliberate selection remains available.

Another Important information: Our project challenge a problem many that did not realize such novelty itself is an creativity.

What it does

Gaze offers a spatial reference to objects in a visual scene, but it also supports ordinary inspection. Jacob’s analysis of eye movement interfaces identifies the ambiguity that arises when looking at an object is treated as a command. In robotic shared control, natural gaze has also been used to complement joystick input for goal prediction, with benefits dependent on the control constraints and timing of available gaze information. These findings motivate a clear distinction between estimating an unexpressed goal and accepting a deliberate command. For an explicit selection interface, the design must specify how an observed object becomes a committed target, how the intended action is authorized, and how users can recognize or revise that state.

Existing systems already combine gaze tracking, object detection, and robot manipulation. GazeGrasp, for example, uses a wearable camera, object recognition, cursor snapping, and dwell activation to support object selection and pick-and-place commands. User studies also show that interpreting the robot’s current task state deserves separate attention. In a Wizard-of-Oz investigation of gaze-driven robotic assistance, Fischer-Janzen et al. report that participants were uncertain whether redirecting their gaze during an ongoing action would interrupt the task. This observation motivates explicit rules for retaining and releasing a selected target, together with feedback that distinguishes browsing, committed selection, and authorized execution.

How we built it

We propose an interaction framework based on explicit target selection, persistent target retention, and configurable action confirmation. Detected objects are presented as candidates in the scene view. The user commits to an object, selects an available predefined action, and authorizes execution through a suitable input channel. The interface retains the selected object’s identity independently of subsequent gaze movement and displays the current target and pending action. Users may deliberately change or cancel the selection. Task completion releases the selection, whereas loss of target tracking pauses execution pending user review.

In this framework, dwell activation and target retention serve different functions. A dwell threshold provides a defined rule for initiating a selection, whereas retention preserves the resulting choice after that input ends. Confirmation can be configured for gaze dwell on a displayed option, a head gesture, or speech, according to the user’s available abilities. The interface is designed to support object and action selection, confirmation, cancellation, and recovery through one usable channel, with additional channels available when appropriate. A library of predefined robot actions handles the execution details supported by the platform, while the user specifies the target and requested operation.

Challenges we ran into

Insufficient torque with an unknown payload Our 2-link arm lacked the power to reach certain positions. We modeled the two-linkage system, computed the failure angle at which torque is exceeded for an unknown mass, and formulated an optimal control problem over joint angles so the robot plans around positions it cannot reliably hold.
Fast arm motion made objects slip during transport Once the gripper held an object, rapid acceleration and deceleration of the arm produced inertial forces larger than the grip friction could resist, so objects slipped or dropped mid-motion. Because the payload mass is unknown, we could not set one fixed safe limit.
Looking is not choosing (the “Midas touch” problem) Gaze is also how people simply inspect a scene, so treating “look” as “command” causes accidental picks. We separated three functions: a dwell threshold to initiate selection, retention to keep the target after gaze moves, and an explicit authorization step before the robot acts. Tuning dwell time was a real tradeoff between false selections and user fatigue.
Hardware integration under time pressure We assembled the robot, wired the electronics and power, and connected camera, gaze pipeline, and motor control in a single weekend. As a last minute change, we wanted to add roll to the hand motion, so we improvised with 3d printed pieces we already had, drilled new holes, and put screws in.

Accomplishments that we're proud of

We made a 4 servo robotic arm that can extend and roll the hand. It can grip objects, and just barely support its own weight.

What we learned

We gained skills in cadding, 3d printing, and improvising solutions to the various lack of tools and materials we had. For example, we ran out of double sided male connector wires, so we used pieces of wire to use female connectors as male ones. The forearm was too long for the mg996r servos to support, so we had to shorten it, improvising screw placements and combinations of out plastic connectors.

What's next for Gaze2Grasp

We would keep improving the eye tracking and interface with the arm. We would need stronger servos, use smooth trajectories, improve grip, and organize everything. Baselines will include established selection policies and input methods that participants can use. Evaluation will examine active input duration, repeated selections, unintended target changes, recovery effort, and comprehension of the current target and action, alongside task completion time and success.