HAPTOS

The surgery that doesn't wait for the surgeon

Comment

Inspiration

We wanted to build something in healthcare that could make a real difference in people's lives. While we were brainstorming, one problem kept coming back. There are lots of places where a human doctor just can't get to you, like remote villages, disaster zones, ships at sea, and research stations. However, a small device you can carry with you can get there. This year's HackHarvard theme was also "Hack to the Moon," so we followed the idea as far as it goes. Mars is about 12.9 light-minutes away, so no surgeon on Earth could steer a robot there in real time. A robot that far from help has to check the tissue for itself and refuse to cut unless it's sure, all with minimal guidance from a remote surgeon. HAPTOS is our first step toward that.

What it does

HAPTOS (Haptic Autonomous Probing & Tissue Operation System) is a voice-driven surgical arm that classifies tissue by touch and only cuts when a confidence check written into the code passes. A surgeon says something like "cut the tissue." Grok Voice hears it and passes the request to Claude, which plans the steps. Claude moves the arm across three stations and probes each one with an impedance sensor. A k-nearest-neighbors classifier labels each reading as bone, fat, tissue, or skin, along with a confidence score. The arm cuts only if the latest probe at that station matches the requested with at least 70% confidence. If no station matches, nothing moves. Ask it to "cut skin" and it answers "skin was not found." Every probe and result is read out loud. A live dashboard shows the arm, the classifications, the impedance readings, a log of every command, and the current Earth-to-Mars light delay.

How we built it

Chinmay led the hardware and Krishnam led the software. We worked together on connecting the two, on testing, and on building the fake tissue samples (phantoms).

Voice: We used the xAI Grok Voice realtime API over a WebSocket connection, with voice activity detection on the server. Grok has one tool, which passes the surgeon's request to Claude word for word. Planner: Claude Sonnet 4.5 runs through Anthropic's API with tools like resolve-tissue, probe, cut, move-to, say, and home. It probes station 1, then 2, then 3 until one matches. Controller: A Python process carries out every tool call Claude makes. It runs the arm, the sensor, the classifier, the voice narration, and the console. The safety check lives here in code, not in a prompt. Sensing: An AD5933 impedance analyzer runs at 30 kHz, with two steel pick electrodes, a DC-blocking capacitor, and an optional force sensor. An Arduino Uno sends readings about 10 times a second. For each probe, the arm settles for 1 second, then takes the median of 15 readings. Classifier: We used scikit-learn's k-nearest-neighbors (k = 5, weighted by distance). Its inputs are the impedance on a log scale, the phase, and the probe force. We trained it on about 30 samples per class: wood stands in for bone, tap water for fat, salted gelatin for tissue, plus live skin. Arm: A Cokoino 4-servo acrylic arm runs on an Arduino Nano with a servo shield. We rewrote its firmware from scratch on top of the standard Servo library, using a simple serial protocol and a calibrated home position saved in memory. Narration and console: Claude's spoken updates use xAI text-to-speech so the surgeon hears each step as it happens. The console sends live updates to the browser and pulls Mars distance data from NASA JPL Horizons. It also has a text command bar and a mic mute button for loud rooms.

Challenges we ran into

Fake tissue turned out to be really hard. Our first phantoms (store-bought fake tissue, a stress ball, and wood) all looked the same to the sensor because they're all insulators. We had to rethink the whole approach around how well each material conducts electricity. We had no measuring tools, so we remade the gelatin recipe in a cereal cup. And we spilled the water too many times to count (and almost fried all of our electronics).

The sensor came with no documentation. We had to figure out on our own that its two power connectors both need power, or the amplifier stays dead. Readings in water dropped to almost zero after a second, and it took us quite a bit of time to realize it was because it was touching the plastic of the cup that we had the water in, and not an issue with the sensor.

The hardware also took some hits along the way. Our Arduino Mega died partway through the build, probably from a 5 V short while we were moving wires, so we moved the whole sensor setup to a backup Uno at night. Then a spliced power wire shorted, and the Uno got stuck restarting over and over. We fixed that too.

The arm was the longest part of the build. Its two middle servos only buzzed. We checked the batteries, the voltage regulator, and the servos themselves, and even wrote a test program to measure the voltage while the arm moved. Eventually we realized that the weight of the electronics on the back of the arm was too heavy for the servos to lift. We had to improvize and only hold our arm in place instead of allowing it to have more freedom.

The software had its own problems. At first Grok started hearing itself, and kept repeating the instructions back to itself, creating an endless loop. We had to lower the volume of Grok to fix it. We also had to get Grok to stop retrying commands after a refusal, and to stop asking "which tissue?" when "tissue" was the tissue type.

Accomplishments that we're proud of

Our proudest moment was when the whole pipeline connected for the first time. Our voice reached the AI, the AI moved the arm, and the arm probed and sensed the tissue. After a weekend of dead boards, shorted wires, and buzzing servos, seeing every piece work together end to end was the best part of the hackathon. We built two microcontrollers, a custom sensor setup, and a three-model AI system into a working demo in about 25 hours.

What we learned

Electrical impedance can tell materials apart surprisingly well, but only if you control the electrode spacing, the contact force, and stray DC voltage. Read the mechanical drawings before you assume a part is broken. Always keep a backup board around, and strain-relieve your wires.

The biggest thing was how to combine the hardware and software so that they could to talk to each other, and how to combine all the different parts of our pipeline into one single project.

What's next for HAPTOS

Measure across more frequencies and collect more training data from realistic tissue phantoms. Build a stronger arm with more range of motion and a real cutting tool. Add haptic feedback so a surgeon far away can feel what the probe feels.

Built With

Share this project:

Updates

Submission history