Zer0Sh0t

Inspiration

More than 2.5 billion people need an assistive product, yet access remains limited, with affordability a major barrier. For people with spinal injuries, muscular dystrophy, limb loss, or vision loss, everyday tasks like picking up a fork can take real effort.

Robotic arms could help, but adapting them to new tasks often requires specialized programming or task-specific training. Someone’s needs change every day, and no one can anticipate all of them. So we asked: what if you could just ask?

What it does

Zer0Sh0t is a voice-controlled robotic arm that combines camera perception with AI planning. Ask it to fetch a supported object or sort colored blocks, and it translates the request into physical actions.

The zero-shot idea is that supported requests do not require task-specific demonstrations or retraining. Instead, pretrained models and our control software work together to interpret requests and execute existing motion skills in new combinations.

Our current prototype operates within a calibrated tabletop workspace and manipulates measured props. It can also use Gemini to name unfamiliar objects, although recognizing an object does not automatically mean it can safely grasp it.

How we built it

We started with a low-cost robotic arm from a previous project and replaced its task-specific training workflow with live camera perception, generative AI, and calibrated motion controls.

We integrated Google’s Gemini Robotics-ER 2 Preview through the Gemini API. Gemini analyzes a camera image and returns descriptive object labels and image locations. Our local vision system connects those descriptions to tracked objects, while calibrated geometry and a measured prop catalog provide the information needed for manipulation. This separates understanding what an object is from deciding how the arm can move around it.

ElevenLabs transcribes spoken requests. A local command router handles common instructions, while Claude handles open-ended requests by selecting objects and calling tools such as picking, placing, and pointing. These tools use object IDs and predefined destination zones. Our control software translates them into physical movements using calibrated coordinates, inverse kinematics, and motion checks.

Together, these components let users request supported actions without writing a new script or training a new motion policy for each request.

Challenges we ran into

Calibration and physical precision were the hardest parts. Small errors in camera alignment, arm geometry, or grasp position could turn a reasonable plan into a missed pick or inaccurate placement.

We measured the workspace, standardized the setup, and repeatedly tested the connection between perception and motion. We also encountered camera-processing bottlenecks and false stops. Getting the arm to understand a request was only one part of the problem; making it execute that request consistently required much more iteration.

Accomplishments that we're proud of

We brought perception, language, and physical control together on a working robotic arm. Our recorded hardware tests include fetching a cube to a handover pad and completing a two-cube color sort.

Seeing an instruction become a physical action made the project’s potential tangible. Reliability still needs work, but we demonstrated the core approach: combining pretrained models with reusable motion skills instead of training a separate policy for every supported request.

What we learned

Zero-shot prompting can reduce the work needed to express a new task, but it does not remove the need for calibration, capable hardware, or reliable control.

We learned that the hardest part is often handing an AI-generated plan cleanly to physical motion. An object label is not a grasp plan, and a sensible sequence of actions is not a guarantee of successful execution. Measuring outcomes and handling failures matter as much as interpreting the request.

What's next for Zer0Sh0t

Our immediate priorities are improving grasp and placement accuracy, reducing perception latency, restoring stricter motion checks, and measuring repeatability across many hardware trials. We also want to make voice stopping reliable while the arm is moving and validate its response time.

As the prototype becomes more reliable, we want to work with people who have spinal injuries, muscular dystrophy, limb loss, or vision loss. Their feedback should guide which tasks we support and how people interact with the arm.

Longer term, we want to expand the range of objects and actions it supports, then work toward requests like “set the table” or “tidy this desk.” Those goals require better manipulation and planning capabilities. We aim to preserve the same interaction: ask for help in ordinary language, without needing to program the robot for each task.