Inspiration
I kept thinking about how every 3D tool I've ever used makes you control a three-dimensional thing through a completely flat input, a mouse, or two fingers on glass. That always bugged me a little. So I wanted to try the opposite: wear something, move your hand, and the model just responds. The Tony Stark hologram thing was the first idea, not gonna lie. But the more I built it the more I realized the "point at things with your hand" part isn't just a cool demo trick, anatomy classes already color-code structures so students can tell them apart, and surgeons genuinely can't touch a screen mid-procedure. So it turned into something that's actually useful for teaching and clinical stuff, not just a flex.
What it does
There's a puck you wear: accelerometer, ultrasonic distance sensor, color sensor, all wired to an ESP32 — and it streams to an iOS app over its own WiFi. Tilt it, and the model rotates. Move it closer or farther from something, and it zooms. Touch it to a real object and the model can pick up that color (as long as you hold it steady for a few seconds. I didn't want it flickering every time the reading jittered).
On top of that the app watches your other hand through the camera, point to aim at something, pinch to grab it, open your palm to blow a model apart into its separate pieces and drag the whole thing around, make a fist to bring it back together, and it's also just listening the whole time for stuff like "reset," "explode," "label this left ventricle," "what color is this." It talks back after basically everything so you actually know it heard you, since there's no screen you're supposed to be looking at.
You can also import your own models, and it remembers them, say the actual name of a file you've already brought in and it'll load it by voice, no digging through a picker.
How we built it
Firmware's C++ on an ESP32, hosting its own WiFi network and pushing telemetry out over WebSocket about 45 times a second. Pitch/roll come from blending the accelerometer and gyro together (a complementary filter, basically averaging the two so it's not jittery but also not laggy). Color comes off a TCS3200, which is honestly a weird sensor, it doesn't give you a number, it gives you a pulse of a certain width, and you have to time it yourself and invert it.
The app side is Swift, SwiftUI, SceneKit for the actual 3D rendering. Hand tracking runs on Vision's hand pose detector, and I built the gesture classifier myself off of comparing where each fingertip is relative to its own knuckle. Voice is all on-device through Apple's Speech framework, and it talks back with the built-in speech synthesizer.
Challenges we ran into
Honestly this is most of what I'd want people to actually read, because almost none of these were obvious until I saw them happen live.
The color sensor flat-out refused to see black. Took me a while to realize why it was using its own light reading to guess whether something was even touching it, but a black surface reflects basically no light, same as pointing it at nothing. So it treated "touching something black" exactly the same as "touching nothing." White worked fine the whole time, which made it more confusing, not less.
I almost fixed that by using the physical distance sensor instead, except that sensor was already doing zoom, and if I'd wired it into the color logic too, it would've silently broken zoom the second color detection kicked in. Caught that before I actually built it, thankfully.
Importing a model sometimes just... didn't replace the default shape. Turned out there was a leftover reference to a property that didn't even exist in the code; it never errored, it just quietly did nothing, so the old shape stayed on screen underneath whatever I'd just imported.
Selecting one part of a multi-part model would recolor every part. Files that split into separate pieces often still share the literal same material object between all of them, not just the same material name, so touching one part's color touched all of them at once. Had to explicitly copy the material for every part when the file loads.
The explode animation was invisible on some files and completely wild on others, because I was moving each piece by a fixed distance, and different files use wildly different coordinate scales. Fixed it by scaling the explode distance to each model's actual size instead of a flat number.
Voice was the worst one, honestly. I added a "Friday" wake word so it wouldn't react to every random word, and that broke things in a way that looked totally random. Turns out if you pause even a little after saying "Friday," the speech recognizer treats that as a finished sentence on its own, so whatever you say next never has "Friday" in it and gets ignored. Took actually setting breakpoints and watching the raw text come in to catch that. Eventually just dropped the wake word entirely; it cost more reliability than it saved.
And then once it started talking back to confirm commands, it got stuck listening to itself, heard its own voice through the mic, transcribed it, and re-triggered whatever it just said, forever. Had to mute the mic while it's actually speaking.
Also spent a chunk of time realizing most "segmented" anatomy models online aren't actually segmented; a bunch turned out to be one single mesh, and one was even AI-generated (you could tell from how the internal file was named). The ones that actually had separate parts came from files meant for multi-material 3D printing, not from anywhere obviously "3D model site" shaped.
And separately, the physical accelerometer is still giving me trouble: it reads all zeros right now, which points to a wiring issue I haven't fully chased down yet.
Accomplishments that we're proud of
Getting real sensor data, not a mouse, not touch, to actually drive a live 3D scene reliably, and getting three totally different input methods (the physical puck, hand gestures, voice) to work at the same time without stepping on each other.
What we learned
Most of the bugs here weren't things I could've caught by just reading the code back. A black surface and empty air can look identical to a cheap light sensor. A wake word sounds simple and isn't. Stuff that seems small. One highlight color, one drag gesture, can quietly break something else you didn't expect it to touch. Had to actually run it, over and over, to find any of this.
What's next for HoloSmith
Getting the accelerometer wiring sorted, saving custom part labels so they survive a relaunch, and looking into using iOS's actual Eye Tracking accessibility feature so you can select a part just by looking at it instead of pointing.
Log in or sign up for Devpost to join the conversation.