Inspiration
Firefighters, police and rescue teams still go through doors blind. They don't know who's inside, where they are, or whether they're moving. Thermal cameras only help once you're already looking into the room. We wanted to know who's on the other side before anyone opens the door.
What it does
Spectre lets you see through walls. Ordinary webcams on the far side of a wall track people. Spectre fuses those views into one live 3D map, and a Meta Quest 2 draws each person as a skeleton, exactly where they're standing, through the wall.
Spectre only ever shows skeletons, never faces or video, so privacy is built in.
How we built it
| Layer | What we used |
|---|---|
| Capture | USB webcams, one laptop per camera, read with OpenCV in Python |
| Detection | A Python camera worker per camera running [pose model] to find people and joints |
| Calibration | Four fixed markers give each camera's position in a shared 3D frame, rechecked every minute |
| Transport | WebSockets: camera workers → fusion server → headset |
| Fusion | A server laptop combines whichever cameras are live, weighted by trust |
| Viewer | Three.js and WebXR on a Meta Quest 2 |
The key idea is that the cameras don't need to see what the viewer can't. A camera on the other side of the wall sees the person. Because every camera is calibrated into the same 3D space, we can put that person exactly where they are in the headset.
Neptune North: failing gracefully
Our rule: when a camera drops out, Spectre gets less complete, never wrong. We built this at two layers.
1. Every camera checks whether it can still be trusted. Each camera rechecks its calibration markers every minute:
| Markers seen | Where expected? | What Spectre does |
|---|---|---|
| All 4 | Any | Recalculate the camera's position, full trust |
| 3 | Yes | Keep the position: one marker is just blocked |
| 3 | No, moved | Remap from 3, and lower its weight if marker error is high |
| 1 or 2 | Yes | Keep the position and keep serving |
| 1 or 2 | No, moved | Drop the camera until the next check |
| Feed frozen | n/a | Treated as disconnected, never stale data |
A camera that can't prove where it is sits out, exactly as if it were unplugged. Showing someone in the wrong place is worse than not showing them.
2. No camera laptop can take the headset down. Our first setup had Laptop B running Camera B and hosting the headset app, so losing it blanked the headset. Now every camera laptop does one job, and the headset connects straight to the fusion server. Lose either camera laptop and the headset carries on with the other camera.
| What happens | First version | Now |
|---|---|---|
| Laptop A drops | Carries on with Camera B | Carries on with Camera B |
| Laptop B drops | Headset goes dark | Carries on with Camera A |
| A camera is knocked | People appear in the wrong place | Remaps, or sits out until trusted |
Marshall Wace: New Worlds
The room behind a wall is unexplored until someone walks in. Spectre opens it up first.
- New data: no single camera has this picture. It only exists once you fuse partial, noisy views, weight each by how far it can be trusted, and drop the ones that can't be.
- A new sense: look at a wall and see who's behind it, live, on a consumer headset.
- New ground: with drones and robots as scouts, rooms nobody has entered can be mapped before a person steps inside.
Challenges we ran into
- Keeping every camera's calibration consistent, and deciding what to do when a camera has moved
- Finding and removing the single point of failure we'd built in by hosting the app on a camera laptop
- Turning "fail gracefully" into hard rules we could test by unplugging things
Accomplishments that we're proud of
- Live fused skeletons from multiple cameras, rendered through a wall in VR
- You can unplug a camera laptop mid-demo and the view stays correct
What we learned
Graceful failure is a design rule, not a feature you add at the end. "Less complete, never wrong" shaped every decision, from calibration to network layout.
What's next for Spectre
- Autonomous scouts: run camera workers on recon drones and ground robots, located by their own onboard tracking instead of wall markers. A drone that loses signal drops out like an unplugged camera.
- Real walls: in our demo the wall is a blanket. Next is adding sensors that pass through walls, like mmWave radar or Wi-Fi sensing. The pipeline doesn't care what the sensor is.
- Backup server: a standby, so the fusion server stops being the last single point of failure.
- Show the gaps: show each camera's coverage in the headset, so a blind spot looks blind, not empty.
- Faster recalibration: trigger a check the moment a camera's markers drift, instead of waiting for the next minute.