InSight

InSight for those without sight. We turn any picture into 3D relief that blind people can feel, drawn by a robot arm with a 3D pen, from family photos to pill bottles to Halloween decorations.

Comment

Inspiration

I wave hello to my grandma. She doesn't wave back. I run up and hug her. Only then does she finally recognize me.

My grandma has severe vision loss, like millions of people around the world. She knows the world through touch: a hug, a familiar texture, the shape of something in her hands.

But most of the visual world is smooth. Family photos, maps, medicine bottles, and food packaging carry their information as printed images. The fingertip feels nothing.

Braille helps, but it only gives the name of a thing, not what it looks like.

So we built InSight: any image, turned into a raised relief you can read with your fingertips, drawn directly onto the object it belongs on.

What InSight Does

InSight takes an image and a target surface (plane, cylinder or sphere), converts the image into a multi-level tactile relief, and uses a 7-DOF Flexiv Rizon 4s with a 3D pen to draw that relief onto the surface. This relief lets blind and low-vision people feel what the image looks like: its shape, its parts, and how they fit together.

A traditional 3D printer can only build upwards on a flat print bed. Our robot arm, in contrast, can place a 3D pen in almost any orientation, which allows for drawing on irregular surfaces like curves.

How we built InSight

InSight's pipeline runs in six stages across 8 ROS2 packages.

  1. Image -> Tactile Relief. Classical CV to read the major edges from the image. These edges become tactile ridges for fingers to read.
  2. Surface Mapping. Convert flat image coordinates into a 3D point. Cylinder uses image wrapping, sphere uses stereographic projection.
  3. Motion Planning. Trajectory generation in Cartesian space with clearance checks. Pen stays within 35 degrees of surface normal, velocity smoothed at kinks, and pen avoids surface collision between strokes.
  4. Controller Execution. Switching between joint controller for approach and park, and Cartesian controller for drawing (streaming TCP targets at 1kHz).
  5. Extrusion. Robot’s digital output triggers ESP32, driving servo that presses pen’s extrude button. The firmware enforces debouncing, a safe boot state, and 2 sec press timeout. Extrusion releases before every lift, and a calibrated lead time precedes each stroke.
  6. Preview. Web browser UI shows tactile relief, tool path, IK animation of full arm, D405 camera feed. Real execution stays locked until preview succeeds.

Challenges we ran into

Jerky Cartesian Tracking. Our first prints on curved surfaces had uneven beads caused by jerky Cartesian tracking. Tuning speed, standoff and pen angle helped only slightly. Switching to joint control didn’t fix it either. The real cause was upstream: the Cartesian controller takes the latest target and does not interpolate between targets, causing a series of small jumps. Increasing streaming targets to 1 kHz, which matches the control loop, made the motion smooth.

Calibration Sensitivity on Sphere. The sphere posed a second problem: it curves in every direction, so it's unforgiving of calibration error. On a tile, a placement error parallel to the surface only shifts the drawing sideways. However, on the sphere, an error in the registered center changes that the pen-to-surface gap differently at every point, pressing into the surface on one side and lifting away on the other. We mitigated this via extensive tuning of parameters like drawing speed and clearance. A proper fix would be to measure the sphere’s position directly by touching points around it with the arm.

Accomplishments that we're proud of

Three Surfaces One Pipeline. InSight draws on flat tiles, spheres, and cylinders. We printed a Calhacks venue map, spherical Jack-O-Lantern decoration, sun/moon label on pill bottle, and a bright outline on a family photo.

Full Hardware–Software Stack. Image processing, surface geometry, trajectory generation, ROS2 stack, C++ executor, and ESP32 firmware all run as one system. The whole pipeline can be previewed in simulation before the real arm moves.

What we learned

The image processing, planning, and control have no deep learning. Every stage is geometry or classical control that we can inspect and reason about:

  1. Cartesian vs joint-space control, and when to hand off between them
  2. Why target rate matters when a controller doesn’t interpolate
  3. How calibration errors propagate through a chain of rigid transforms, and why curved surfaces are far less forgiving of them.

What's next for InSight

Surface Scan instead of Surface Model. Currently, each object is described by a predefined surface model: a plane, cylinder or sphere with known dimensions. Next, we would want to scan it directly with a wrist-mounted depth camera. This would let InSight draw on objects of any shape without a CAD model. It would also measure where the object actually sits, which removes the calibration errors we fought on the sphere.

Force-Controlled Contact. Use the Flexiv’s joint torque sensing to hold a constant light contact force against the surface instead of a fixed standoff. This directly addresses the uneven-bead problem on the sphere.

Closed-Loop Extrusion. Currently, extrusion is open-loop: the robot sends a press request but never confirms plastic is flowing. Adding a flow sensor, for instance, would let the robot detect a jam or missing strand, and adjust.

Built With

Share this project:

Updates

Submission history