PokeShape

By only poking, we can reconstruct an object's geometry
More and more poking converges to the ground-truth shape

Can a robot understand a scene with total darkness?

Imagining a robot in a room with total darkness. It can do no planning, no reasoning, no navigation, no nothing, because it has no sense of where the objects are, and how do they look like.

But think about how humans navigate through total darkness. We touch, we feel, and we form a "world model" of object geometries. Through only force feedback from the skin, we reconstruct the entire scene in our mind (though it seems that we represent them latently :( ).

In PokeShape, we demonstrate that understanding a scene requires only 1-DoF pokes, even without our 27-DoF human hands. Using a custom-built, low-cost arm with torque feedback, we reconstruct object geometries near-perfectly, be it circles or rectangles or more, in MuJoCo. This is done only through pokes, like how humans use index fingers to poke on objects. The 3D digital twin created can empower numerous downstream tasks, such scene understanding and manipulation.

Sample-efficient shape reconstruction

PokeShape reconstructs an object through a sequence of targeted probes. The strategy follows how a person might locate and identify an object in the dark: search until contact is established, then concentrate measurements around the object’s boundary. Each subsequent probe is chosen to resolve uncertainty in the current reconstruction.

Contact detection without tactile sensors

The arm operates without cameras or dedicated tactile sensors. Instead, we estimate contact forces from joint deflections under compliant position control. Each joint behaves as a virtual spring: the difference between its commanded and measured position, multiplied by its stiffness, gives an estimate of the joint torque. We then use the arm’s kinematics to estimate the corresponding force at the fingertip.

Contact is detected when the estimated force opposing motion exceeds 1.5 N. The measured joint angles determine the fingertip position at contact, providing a geometric constraint on the object’s surface.

Information from contacts and free space

The probe consists of a 40 mm shaft with a spherical tip. Each probing action moves the fingertip along a straight line until contact occurs or the prescribed travel distance is reached.

A successful contact constrains the surface location. A probe without contact establishes that the traversed path is free space. Both observations contribute to reconstruction: contacts constrain where the object must be, while free-space observations exclude shapes that would obstruct an already traversed path.

Step 1: Locate the object

The initial search sweeps across the center of the search region at a fixed height. If no contact is detected, the sweep is repeated 1 cm lower. Additional sweep locations are introduced by successively subdividing the search region, first at the quarter positions and then at the eighth positions. The vertical spacing accounts for the wrist’s 15 mm offset above the fingertip.

After the first contact, the arm probes from the opposite direction. Further probes cross the midpoint between these contacts and follow both diagonals. The resulting eight contact points provide an initial polygonal approximation of the object’s outline.

Step 2: Refine the boundary

We refine each segment of the initial outline by probing inward along its perpendicular bisector. Rather than assuming that adjacent contacts belong to a straight edge, we consider both straight and curved continuations of the boundary. The new measurement determines how the segment is treated:

If the contact agrees with the straight continuation and distinguishes it from the curved alternative, the segment is classified as straight within the reconstruction tolerance.
If the straight and curved predictions are too similar to distinguish at the current resolution, the segment requires no further refinement.
Otherwise, the new contact divides the segment into two, and both are examined recursively.

Corners are checked separately. When two fitted straight edges intersect, an additional probe is directed toward their predicted intersection. Contact near that intersection supports a sharp corner; contact before it supports a rounded transition and constrains its radius. If the boundary measurements instead consistently fit a single circle, the outline is represented as circular.

This procedure accommodates boxes, cylinders, rounded boxes and capsule-shaped objects without requiring the object category in advance.

Step 3: Estimate height

Four downward probes—one near the center and three distributed toward the boundary—estimate the object’s height and assess whether its top surface is approximately planar. Under the assumption of vertical walls and a flat top, the measured outline and height define the three-dimensional reconstruction.

Maintain a belief over possible shapes

Alongside the measured boundary, PokeShape maintains a belief over candidate shapes: boxes, cylinders, rounded boxes and capsules with varying dimensions and orientations. After each probe, candidates are scored against all observed contacts and free-space constraints. A candidate is penalized if it predicts a surface inconsistent with a contact or occupies space through which the fingertip has already passed.

We visualize this uncertainty using 18 sampled hypotheses, shown as faint wireframes. Their overlap produces an approximate occupancy map on the table, displayed in purple. Early observations may support several competing explanations; subsequent probes constrain the dimensions, orientation and shape until the hypotheses concentrate around the reconstructed object.

How sample-efficient is it?

We ran 36 trials in MuJoCo (6 shapes × 6 placements), with simulated sensor noise and joint friction:

Object	Pokes (median)	Outline error (worst)	Height error (worst)
Cylinder, Ø40 mm	18	0.49 mm	0.1 mm
Cylinder, Ø60 mm	22	0.39 mm	0.1 mm
Cylinder, Ø90 mm	37	0.42 mm	0.1 mm
Box, 90 × 56 mm	32	0.93 mm	0.1 mm
Rounded box, 90 × 60 mm, 15 mm corners	30	1.07 mm	0.1 mm
Pill, 100 × 50 mm	32	1.02 mm	0.1 mm

For comparison, poking straight down on a 5 mm grid over the same 18 × 17 cm search box takes about 1,200 pokes, and still only pins the outline down to ±2.5 mm. PokeShape gets within about a millimeter in 20 to 40. We provide only simulation results here because the error is easier to measure.

Example of technical difficulties

With no touch sensor, a "touch" is just the joints being pushed off target, and the arm pushes itself too. The arm's own inertia and friction read as fake touches in mid-air. The probe is so short that the wrist hangs 15 mm above the fingertip and hits objects first. Our algorithm will work better for longer pokes, but due to 3D printing constraints, we are unable to test that now.

Takeaways

Touch alone is enough. One stick and straight-line pokes rebuild an object to about a millimeter in 20 to 40 pokes. Misses matter as much as hits, assuming less and confirming more beats clever priors, and keeping a belief of what the object could still be gives the robot a world model you can actually watch converge.

Disclaimers

The custom arm was built before the Hackathon and brought to here. A low level controller system came with the arm. It is no different from other open-sourced developer-style arms like SO-101, but only using motors that can provide torque feedback (our arm uses the same motors in YAM). Everything other than the arm and its controller system is done during CalHack 13.0.