Inspiration
I've wanted to build an R2 unit since I was a kid.
Not a replica — a working one. Something that rolls into a room on its own, figures out what's going on, and does something about it. That's the part of R2 that always got me: he was never the hero. He was the one who showed up, assessed the situation, and handled it while everyone else was panicking.
The medical angle came later, and it came from a simple observation. In a hospital at 3 AM, or a field tent after a bus crash, the thing that's short isn't medicine or equipment. It's attention. Somebody has to walk the room and notice which patient is going quiet. When there are twelve patients and two nurses, somebody doesn't get noticed — and that's not a failure of skill, it's a failure of arithmetic.
That's a problem a droid is actually good at. Not diagnosing. Not treating. Just going to every patient, every time, and taking a reading. Being the thing that doesn't get tired and doesn't skip anyone.
That's where the name came from. R2 Hope — the droid that shows up.
What it does
R2 Hope is an autonomous triage-support robot. It automates the part of triage a person currently has to do on foot: finding the patient, getting to them, and taking a reading.
In autonomous mode it sweeps the room looking for a patient — not just any person, but specifically someone in a medical gown, identified by combining YOLO person detection with a calibrated color signature. A person alone isn't enough and the gown color alone isn't enough; both have to pass before it commits. Once it has a lock, it drives toward them, steering around obstacles with lidar and slowing as it closes, then holds position on arrival.
From there it switches from navigating to monitoring. Using facial landmarks it tracks eye closure against a baseline it calibrates on the spot for that specific person, and reports PERCLOS — the share of a rolling 60-second window the patient's eyes were closed. PERCLOS is the standard clinical measure of drowsiness, and it's the earliest honest signal you can get from a camera that a patient is fading.
It also has a complete manual mode: mixed throttle-and-steering keyboard teleop over UART, so an operator can take over at any moment.
What it deliberately does not do is make the call. It gathers the data. The clinical judgment stays with the human.
How we built it
Split the brain
Before we cut a single wire we spent a long time arguing about architecture, and almost every decision from those conversations is why the robot works at all.
The big one: a Raspberry Pi 5 is the navigator, an ESP32 is the driver.
The Pi runs the camera, the lidar, and the vision models, making maybe 5–10 decisions per second. The ESP32 does nothing clever — it runs motor control at a hard 100 Hz, forever.
Those two jobs have incompatible requirements. The Pi runs Linux and Python; it can stall for 200 ms because an inference ran long or the scheduler felt like it. That's fine for deciding where to go. It is not fine for holding a PWM signal. So we gave the motors a processor that cannot be interrupted by a YOLO frame.
They talk over UART with a line protocol we specified ourselves — plain ASCII, one reply per command:
M <left> <right> per-wheel PWM -> OK <l> <r> <heading>
R <degrees> closed-loop gyro turn -> TURN ... DONE
S stop -> OK 0 0
P ping -> PONG
G orientation -> H <z> PITCH <x> RATE <r>
And the single most important feature in the project lives on the ESP32: a watchdog. If no motor command arrives for 500 ms, the motors stop. The Pi sends a heartbeat at 10 Hz to keep it fed. So if Python crashes, if SSH drops, if I close the laptop — the robot stops. It does not keep driving. We designed the failure mode before we designed the feature.
Finding the patient
A person is not a patient. That distinction is the entire detection problem.
We run YOLOv11n for person detection, then check an HSV color mask inside each box. The patient is the intersection, and it has to clear two independent gates:
$$ \text{patient} \iff \left( N_{\text{px}} > N_{\min} \right) \;\wedge\; \left( \frac{N_{\text{px}}}{A_{\text{box}}} > f_{\min} \right) $$
where \( N_{\text{px}} \) is the matching pixel count and \( A_{\text{box}} \) the bounding box area, with \( N_{\min} = 800 \) and \( f_{\min} = 0.04 \).
Why two gates? Because a count alone fails in both directions. A patient far away is small and fails a fixed threshold. A bystander standing close fills the frame, and a few hundred stray pixels from a lanyard sail right past it. The fraction gate kills the bystander; the count gate kills sensor noise. Together, a close-up person with 1,200 stray pixels in an 80,000 px box (\( 1.5\% \)) is correctly rejected, while a real gown passes at \( 25\% \) whether near or far.
Monitoring fatigue
Parked at the patient, we compute eye aspect ratio from facial landmarks — eye height over eye width:
$$ \text{EAR} = \frac{\lVert p_2 - p_6 \rVert + \lVert p_3 - p_5 \rVert}{2\,\lVert p_1 - p_4 \rVert} $$
Every face has a different resting EAR, so an absolute threshold is useless. We calibrate a per-person baseline \( \text{EAR}_0 \) during the first seconds of eyes-open and count a frame as closed when \( \text{EAR} < 0.75\,\text{EAR}_0 \).
PERCLOS is then the proportion of closure over a rolling window:
$$ \text{PERCLOS} = \frac{1}{T}\int_{t-T}^{t} \mathbb{1}!\left[\text{EAR}(\tau) < \kappa\,\text{EAR}_0\right] d\tau, \qquad T = 60\text{s} $$
We chose this deliberately over the flashier option. We could have tried to read pain or distress from facial expression. We decided not to, because it doesn't work reliably and we weren't willing to demo something we couldn't defend.
Not hitting things
The lidar gives ~210 scattered \( (\theta, r) \) points per revolution, which is awkward to reason about. We bin them into 36 sectors of \( 10° \) and keep the nearest return in each:
$$ d_i = \min {\, r_k : \theta_k \in S_i,\; r_k > r_{\min} \,} $$
Nearest, not average, on purpose — for not crashing, the chair leg at 40 cm matters and the wall 4 m behind it does not.
On top we built a cut-down Vector Field Histogram, the approach used in F1TENTH autonomous racing. Score every candidate bearing, steer toward the best:
$$ \text{score}(i) = c_i - \beta\,\lvert \alpha_i \rvert, \qquad c_i = \min_{|k| \le w} d_{i+k} $$
Two details matter. \( c_i \) is the corridor width, not the distance at \( i \) alone — the robot has physical width, so the sectors beside a gap must be clear too, or it aims at an opening it can't fit through and clips the edge. And \( \beta \) is a straight-ahead bias costing score per degree off center; without it the robot drifts sideways forever just because the room is marginally roomier that way.
That becomes a steering correction mixed into the wheels via standard differential drive kinematics:
$$ v = \frac{v_R + v_L}{2}, \qquad \omega = \frac{v_R - v_L}{L} $$
In practice: left = speed + turn, right = speed - turn. Smooth arcs, no pivoting.
The body
Chassis, head, legs and caster mounts designed in Fusion 360 and 3D printed. We also built a Webots simulation world so we could break the navigation logic in software before breaking it on hardware.
Challenges we ran into
Honestly? Everything broke. Hardware, soldering, firmware, software, and the laptop running the IDE — in roughly that order of pain.
The robot kept falling over backwards
This was the big one, and the fix taught me the most.
My instinct was to solve it with the gyro. It's an orientation sensor, the robot has an orientation problem, done. That instinct was wrong, and understanding why was the most valuable thing I learned all weekend.
R2 Hope rolls on two driven wheels plus a caster. It's statically stable — not an inverted pendulum, so there's no balancing to be done. By the time the gyro says "you are tilting," you are already falling. A gyro cannot save a statically stable robot. It can only file a report.
The actual cause was torque reaction: commanding full PWM instantly makes the wheels shove hard against a chassis with a high center of mass, and the body pitches backward. The fix was a slew-rate limiter in firmware — PWM can only change a few counts per 20 ms tick:
$$ u_{t+1} = u_t + \text{clamp}\big(u^{*} - u_t,\; -\Delta_{\text{down}},\; +\Delta_{\text{up}}\big) $$
with \( \Delta_{\text{up}} = 4 \) and \( \Delta_{\text{down}} = 12 \) at 50 Hz — about 200 PWM/sec up, and faster to stop than to go. The robot now accelerates instead of lurching. That's what stopped the falling. Not the gyro.
The gyro then became a genuinely useful last-resort guard, but we had to trigger on rate, not angle. Our IMU is mounted high and off-center, and an accelerometer fundamentally cannot distinguish gravity from linear acceleration — so its tilt estimate is corrupted at exactly the moment you're accelerating, which is exactly when you need it. Angular rate is immune. We cut the motors at \( 90\,°/\text{s} \) of pitch rate.
Soldering and wiring
Hours went into connections that looked fine and weren't. Motors that ran one direction only. A motor that ran backwards — fixed by swapping IN3 and IN4 in firmware rather than redoing wiring at 2 AM. Joints that passed a continuity check on the bench and failed under vibration once the robot was actually moving.
One bad sensor bricked the whole robot
After one fall, R2 Hope went completely dead. No PONG, no motor response, nothing. We traced it to the MPU6050 being knocked loose on impact, which made mpu.begin() fail, which hit this:
if (mpu.begin() != 0) {
Serial2.println("MPU FAILED");
while (1) delay(1000); // halts forever - kills ALL commands
}
An optional sensor took down the entire control loop, including everything that had nothing to do with it. We rewrote it to degrade gracefully: flag the gyro unavailable, skip the features needing it, keep driving. Never let an optional sensor take down the control loop.
We had a quarter of the control authority we thought
We assumed 0–255 of PWM to work with. We put the robot on the floor and ramped it by hand to check. The wheels don't move at all below 20, and 30 is already medium speed.
The real usable band is roughly \( 20 \)–\( 70 \), not \( 20 \)–\( 255 \). Every gain in the system — steering, turning, cruise — had to be resized for a band a quarter of what we'd assumed. Otherwise we'd have shipped a robot that only knew how to go full speed.
A Raspberry Pi tried to install CUDA
pip install ultralytics cheerfully began downloading 2.5 GB of NVIDIA CUDA libraries — cuDNN alone was 651 MB — onto a Raspberry Pi with no GPU and no room, and killed the SSH session mid-download. Installing CPU-only torch wheels explicitly first, then ultralytics, fixed it and freed 2.7 GB of wasted cache.
Smaller ones that still cost real hours
- Bracketed paste. Pasting a command into a serial terminal wraps it in invisible escape codes (
ESC[200~), silently corrupting every parse. Every command returnedERR bad. Fixed by filtering to printable ASCII and writing the parser by hand instead of trustingsscanf. - A race on the serial port. Our ping kept returning
OK 0 0instead ofPONG— the background heartbeat thread was also reading the port and swallowing replies. Fixed by making the heartbeat the sole reader. - The heartbeat cancelling our own turns. The ESP32 runs a gyro turn to completion, but any incoming
Maborts it — so our own 10 Hz heartbeat killed every turn ~100 ms in. Fixed with a pause flag. - A black video stream. Hours debugging a camera that was working perfectly. The real cause: a stale file. Our deploy silently kept an old copy, so we were debugging code that wasn't running. We now
md5sumboth ends before debugging anything. - Python buffering.
tail -fshowing nothing, because Python buffers stdout when redirected.python3 -u.
Accomplishments that we're proud of
It doesn't fall over anymore. Going from a robot that tipped on every start to one that accelerates smoothly — by understanding the physics instead of adding more sensor — is the thing I'm proudest of.
The safety architecture. The watchdog, the slew limiter, and a panic stop that ignores direction entirely all sit below anything clever. Nothing in the Python can override them. If the brain dies, the body stops. That's a property we designed in from the first hour, not patched in after a crash.
Detection that actually discriminates. Getting from "anyone standing nearby is the patient" to a two-gate test that rejects a bystander at 1.5% coverage while accepting a real gown at 25% near or far — with the math to explain why — felt like real engineering rather than threshold-fiddling.
We shipped continuous steering, not stop-and-turn. Most hackathon obstacle avoidance is: see wall, stop, spin, go. Ours scores every bearing and arcs around things while still moving. It looks like a vehicle, not a Roomba.
We tested instead of assuming, once, and it saved us. The lidar started returning points at 0.11 m and I was certain it was seeing its own mounting bracket — ready to filter out everything under 25 cm. A teammate pushed back: he thought it was just his hand near the sensor during the test.
So we looked at the data instead of arguing. Across 618 scans, the close returns were smeared across \( +165° \), \( +95° \), \( +175° \), \( -15° \), \( -5° \) — everywhere. Fixed structure would appear at a fixed bearing. It was a moving hand.
Had I applied that filter, I'd have permanently blinded R2 Hope to anything within 25 cm, to fix a problem that didn't exist. Best 20 minutes we spent all weekend.
What we learned
Safety in robotics is architectural, not reactive. The watchdog, the ramp, and the panic stop work because they're the lowest layer, underneath everything clever.
Measure, don't assume. The deadband, the lidar artifact, the dead camera stream — every one of them was an assumption that cost us hours and a measurement that cost us minutes.
Know where your system stops. The most important thing we built might be the thing we chose not to claim. We don't diagnose. We don't read pain. We take one honest measurement and hand it to a human.
And the one I keep coming back to: the right answer to "can the gyro stop it falling over?" was no. The ramp stops it falling. The gyro just notices. Getting that wrong first — and then understanding why — was worth more than if it had worked on the first try.
What's next for R2 Hope
- Multi-patient triage. Right now it finds a patient. Logging each one with location, timestamp and PERCLOS, then ranking worst-first, turns a single monitoring pass into an actual prioritized queue — real triage.
- LLM-generated handoff reports from the monitoring data, used as a reporter and dispatcher — deliberately not in the control loop, where its latency has no business.
- A rotating head, so tracking a patient doesn't require turning the whole robot.
- Lower the center of mass. The slew limiter solved the symptom honestly, but the LiPo and power bank belong at the floor of the chassis. No amount of firmware beats that.
- A web dashboard for remote monitoring and voice dispatch.
Built with: Raspberry Pi 5 · ESP32 · Python · OpenCV · Ultralytics YOLOv11 · MediaPipe · RPLidar · MPU6050 · Arduino / C++ · Fusion 360 · Webots · UART
Built With
- 3d-printing
- arduino
- autodesk-fusion-360
- c++
- computer-vision
- esp32
- i2c
- mediapipe
- mpu6050
- numpy
- opencv
- python
- pytorch
- raspberry-pi
- robotics
- rplidar
- uart
- ultralytics
- webots
- yolo
Log in or sign up for Devpost to join the conversation.