-
-
where_am_i
-
flare_recover_verify
-
voice_command
-
tumble_20dps_locked
-
lock
-
starnet_webcam_solve_rate
Inspiration
A spacecraft's computer crashes and reboots with no idea which way it's facing. Real satellites solve this like sailors did: photograph the stars and match the pattern to a catalog, the lost-in-space problem. But spacecraft usually get lost because something went wrong, so they're often tumbling, and every star smears into a streak that standard software can't read. And on Earth, GPS jamming hits ships, aircraft and drones every day. When GPS lies, the sky doesn't.
What it does
- A simulator hides the truth. It draws the real sky (stars to magnitude 5.5, planets from NASA JPL's DE421) at a secret orientation; the tracker only sees the image.
- Star tracker. StarNet, our CNN, finds stars inside motion-blur streaks and glare; tetra3 matches them to the catalog and solves the 3-axis attitude to a few arcseconds.
- Lock you can trust. Lock is staged (detect, match, 6 verification checks over ~5 s), then held while tumbling at 20°/s by coasting through bad frames. When lost, it recovers by itself: detumble, search, verify, LOCK.
- "Where am I?" One button slews the spacecraft along the ecliptic, spots the planets and triangulates its position in the solar system, with an error ellipsoid.
- You play mission control. A console in the browser has a Tumble knob, Solar Flare and WHERE AM I? buttons, an OLED-style status screen and dials for a pan/tilt spacecraft model. Turn it up, blind it, and watch it recover.
- Dashboard. Identified stars and constellations drawn from our estimate, error against the hidden truth, a spacecraft inset with its spin axis, position map, and live switches between star finders (classic / tuned tetra3 / StarNet) and between a clean feed and a simulated webcam view.
- It talks, and listens. Voice callouts by ElevenLabs ("Solar flare! Sensor blinded", "Lock reacquired", "Position fixed from the planets"), and spoken commands via ElevenLabs speech-to-text ("where am I", "tumble at ten degrees per second", "use StarNet").
How we built it
Simulator → frames → StarNet → tetra3 → attitude → planet navigation → dashboard + console.
- Simulator (Python, NumPy, OpenCV, Skyfield): star rendering with motion blur, glare, noise, hot pixels and planets; truth and commands over UDP, frames shared with the solver in memory. The solver never sees the truth.
- Synthetic webcam: since we have no camera rig at this stage, a software camera (keystone, defocus, vignetting, noise, JPEG compression, plus our checkerboard calibration) imitates a webcam filming the screen. One click on the dashboard switches the solver between the clean feed and the webcam view.
- StarNet (PyTorch): 470k-parameter CenterNet-style network, trained on endlessly rendered, exactly labelled frames at 0–20°/s; outputs stars, planets, hot pixels and a glare mask.
- tetra3 (ESA's open-source solver) for star ID; our own acquisition state machine, recovery maneuvers and coasting on top.
- Navigation: least-squares position from 3+ planets against DE421.
- Console: a virtual board in the browser that speaks the same serial protocol as our Arduino firmware, through a Python bridge. Dashboard: Python server + Server-Sent Events. Voice: ElevenLabs text-to-speech and speech-to-text.
Challenges we ran into
- Our Arduino board broke. The console firmware (knob, buttons, OLED, servos) was written, but with no working board we built a virtual console that plays the same protocol, so the demo runs without hardware.
- Tumble breaks classical star finders: at 20°/s tetra3 alone solved 11.5% of frames, 67% hand-tuned. That gap is why StarNet exists.
- StarNet broke on webcam images (recall down to 0.07) until we found the cause was geometry, not noise: warping the image to screen pixels before detection brought it back to 96–100%.
- One bad frame lost the lock. Through the webcam, bad frames come in 0.7 s bursts; coasting at the measured spin rate fixed it.
- Our first real-camera test failed (1,528 "stars" from screen reflections), which led us to build a synthetic webcam to test the whole camera path in software.
- Lag: we moved planet search off the main loop and shared frames between processes (2 → 15+ solves/s).
Accomplishments that we're proud of
- 97% solve rate at 20°/s through our simulated webcam (StarNet + tetra3), vs 51–61% for the best tetra3 setup; 100% on clean frames at every rate from 0 to 20°/s.
- Arcsecond accuracy when steady (median 4–7″), and lock held at 20°/s.
- Position, not just pointing: ~10,000 km in the solar system from planets alone.
- A mission control a stranger can walk up to, talk to and try to break.
What we learned
- Benchmark honestly. On Stellarium frames, StarNet solved only 38% (tuned tetra3: 100%). It learned our renderer; it earns its place under fast tumble, not everywhere.
- The AI-vs-classical gap was in one step, finding stars. Pattern matching was already solved; we just fed it better.
- Live demos are decided by small things: verify before LOCK, coast through dropouts, fix geometry in the right place.
What's next for Lost in Space
- A real camera rig: film the screen in a dark box, then real night-sky images, and fine-tune StarNet on them.
- Earth mode: stars + accelerometer + clock = latitude and longitude without GPS.
- Run on a Raspberry Pi / Jetson, CubeSat-class compute.
- Point at a target and plan a transfer (getting from A to B).
- Rebuild the physical console on a new board (the firmware is ready).