Inspiration
The idea came from a family trip. We watched our parents struggle to take a single decent photo — one of them squinting at the screen, backing up into strangers, trying to get both the whole family and the landmark behind us into frame. Meanwhile my brother and I stood frozen in a pose for two minutes, blocking a footpath while everyone else waited for us to finish.
Tripods don't fix this. A tripod holds a camera still, but composing a shot against a specific background needs someone behind the camera who can move it — and someone telling you where to put your arm. We wanted to build the person behind the camera.
What it does
Auto-selfie is a small autonomous rover that photographs you.
You place it down, wake it with a raised hand, and say what you want out loud:
- "Frame me next to the big tree" → a standard rule-of-thirds composition, you on one third, the tree on the other
- "Make me point at the bird" → the same framing, plus it coaches your arm until you're actually pointing at the bird
It then drives itself into position, tilts the phone, tells you to shift your hand up or left if it needs to, counts you down, and takes the shot. You never touch anything.
The key insight that makes it work: driving the camera and zooming do different things. Moving the rover changes perspective — how big the background looks next to you — while zoom only changes overall size. That splits an intimidating "compose a photo" problem into four independent, controllable axes: turn, drive, tilt, crop.
The second insight: the rover never tracks where it is. It has no wheel encoders and no map. You are its coordinate system. Your shoulder width in pixels is its distance measurement, re-measured fresh in every single frame, so nothing drifts, ever.
How we built it
Three pieces talking over one Wi-Fi network:
Phone (the eye) — an iPhone mounted backwards on the rover, facing you. It runs a single HTML page in Safari, served over HTTPS by the laptop. No app install. The page captures camera frames, compresses them to small JPEGs, and streams them over a WebSocket at ~12 fps, along with gyroscope and gravity readings at 50 Hz. It also does speech recognition and speaks the coaching cues out loud through its own speaker.
Laptop (the brain) — Python. Ultralytics YOLO-pose finds the person and the 17 body keypoints; a tracker locks onto the background object you named. A two-tier parser turns your spoken sentence into a GoalSpec (a fast local keyword grammar first, a language model as fallback). Then a pure control function converts framing error into motor commands. "Pure" is doing real work here: our decision layer imports nothing but math, which meant we could replay logged runs at 10× speed at a desk instead of debugging outdoors.
ESP32-S3 (the muscle) — deliberately dumb firmware. Two L298N drivers running four DC motors in ganged pairs, a 28BYJ-48 stepper on a ULN2003 for camera tilt, and a forward distance sensor. It applies what it's told, reports telemetry back at 20 Hz, and — critically — kills the motors if it hasn't heard a valid command packet in 250 ms.
The pointing feature turned out to be almost free: YOLO-pose already returns wrist and elbow keypoints in the same inference pass, so "are you pointing at that?" is just atan2 twice and a subtraction.
Challenges we ran into
Latency that grows while you watch. Our first build got steadily worse over a session until the rover was reacting to three-second-old video. The cause was frames quietly queueing up in the reader. Fix: a thread that keeps only the newest frame and throws the rest away. Dropped frames are fine; late frames are poison. We lost an afternoon to this.
The L298N deadband. These drivers lose almost 2 V across the bridge, so below roughly 30% duty nothing moves at all. Our proportional controller kept computing perfectly correct small corrections that silently did nothing, and the loop looked dead every time it got close to its target. Every command now gets a minimum-duty floor applied on top.
The 28BYJ-48's backlash. Cheap geared steppers have 1–2° of slop, so the tilt angle depended on which direction you arrived from. Instead of fighting it mechanically, we closed the loop in software: the phone's own gravity sensor tells us the true pitch, so the rover just corrects until the measurement is right. The same trick absorbs missed steps and slip.
Left and right. The phone faces you, so your left is the image's right. We got this backwards on the first run and spent a genuinely funny few minutes watching a tester move their arm further and further in the wrong direction. It now has its own test case.
Safari's security rules. A page needs HTTPS to access the camera — but an HTTPS page can't open a plain WebSocket to a device on your local network. We spent our first hour on this instead of the fun parts, because if it hadn't worked the whole architecture would have needed rethinking.
Coaching that doesn't drive you insane. Our first version spoke every frame and oscillated: "up… down… up…". The fix is all interaction design, not vision — one instruction at a time, a minimum 1.8 s between utterances, hysteresis so it widens its tolerance once you're inside it, and a hard give-up timer. An imperfect photo beats an endless correction loop.
Accomplishments that we're proud of
- It actually works with no encoders, no LiDAR, and no map. Anchoring everything to the subject removed an entire class of drift bugs instead of adding one, and it's genuinely more robust than odometry would have been on a skid-steer chassis that scrubs during every turn.
- No iOS app. A web page on a phone, a laptop, and an $8 microcontroller. Nothing to install, nothing to sign, nothing that expires in 7 days.
- Pose-directed framing. Being able to say "make me point at the bird" and have a robot position itself, then talk you into the pose, is the moment that makes people laugh out loud when they see it.
- A safety story we trust. The stop path doesn't depend on our Python. Pull the laptop's Wi-Fi and the wheels stop in a quarter second — we tested it on video.
- A replay harness. Logging every frame, observation and command meant we tuned control gains indoors at a desk rather than walking back and forth across a field. Best hour we spent.
What we learned
- Reframing the problem beats solving the hard version. Our first design tried to compute a target position in 3D space. Realizing that "drive until the background looks the right size next to you" is a one-dimensional, strictly monotonic search made the whole thing tractable in a weekend.
- Calibrate before you tune. Almost every "bad gain" we chased turned out to be a wrong field-of-view number or an unmeasured motor deadband.
- Keep the decision logic pure. It cost nothing at the time and paid for itself repeatedly — unit tests with plain dictionaries, instant replay, and a clear path to porting.
- A cheap part's weakness is often a software problem. The stepper's backlash, the driver's deadband, and the missing encoders were all solved in code rather than with better hardware.
- How a robot talks to you matters as much as how accurately it measures. A system that nags is worse than one that's slightly less precise and knows when to shut up.
What's next for Auto-selfie
- Full-resolution capture. Our browser frame grab tops out around 1080p–4K with none of the phone's computational photography. A thin native capture path would give us real 48 MP photos.
- Cut the laptop. Port the control logic to run on the phone itself, or move it onto a Raspberry Pi on the rover. The logic is already isolated for this — it's a transliteration, not a rewrite.
- Group shots. Multiple subjects, pose goals per person, and "everyone look here" coordination.
- Reference-photo matching. Show it a photo you like and have it recreate that composition and pose.
- UWB ranging. The iPhone's ultra-wideband chip could give centimetre-accurate distance to the subject that keeps working through occlusion — no vision required.
- Better obstacle sense. A depth camera, or the front-facing camera we currently leave unused, so it can move faster with confidence.
Built With
- esp32
- iot
