Inspiration

Firefighters, police and rescue teams still go through doors blind. They don't know who's inside, where they are, or whether they're moving. Thermal cameras only help once you're already looking into the room. We wanted to know who's on the other side before anyone opens the door.

What it does

Spectre lets you see through walls. Ordinary webcams on the far side of a wall track people. Spectre fuses those views into one live 3D map, and a Meta Quest 2 draws each person as a skeleton, exactly where they're standing, through the wall.

Spectre only ever shows skeletons, never faces or video, so privacy is built in.

How we built it

Layer What we used
Capture USB webcams, one laptop per camera, read with OpenCV in Python
Detection A Python camera worker per camera running [pose model] to find people and joints
Calibration Four fixed markers give each camera's position in a shared 3D frame, rechecked every minute
Transport WebSockets: camera workers → fusion server → headset
Fusion A server laptop combines whichever cameras are live, weighted by trust
Viewer Three.js and WebXR on a Meta Quest 2

The key idea is that the cameras don't need to see what the viewer can't. A camera on the other side of the wall sees the person. Because every camera is calibrated into the same 3D space, we can put that person exactly where they are in the headset.

Neptune North: failing gracefully

Our rule: when a camera drops out, Spectre gets less complete, never wrong. We built this at two layers.

1. Every camera checks whether it can still be trusted. Each camera rechecks its calibration markers every minute:

Markers seen Where expected? What Spectre does
All 4 Any Recalculate the camera's position, full trust
3 Yes Keep the position: one marker is just blocked
3 No, moved Remap from 3, and lower its weight if marker error is high
1 or 2 Yes Keep the position and keep serving
1 or 2 No, moved Drop the camera until the next check
Feed frozen n/a Treated as disconnected, never stale data

A camera that can't prove where it is sits out, exactly as if it were unplugged. Showing someone in the wrong place is worse than not showing them.

2. No camera laptop can take the headset down. Our first setup had Laptop B running Camera B and hosting the headset app, so losing it blanked the headset. Now every camera laptop does one job, and the headset connects straight to the fusion server. Lose either camera laptop and the headset carries on with the other camera.

What happens First version Now
Laptop A drops Carries on with Camera B Carries on with Camera B
Laptop B drops Headset goes dark Carries on with Camera A
A camera is knocked People appear in the wrong place Remaps, or sits out until trusted

Marshall Wace: New Worlds

The room behind a wall is unexplored until someone walks in. Spectre opens it up first.

  • New data: no single camera has this picture. It only exists once you fuse partial, noisy views, weight each by how far it can be trusted, and drop the ones that can't be.
  • A new sense: look at a wall and see who's behind it, live, on a consumer headset.
  • New ground: with drones and robots as scouts, rooms nobody has entered can be mapped before a person steps inside.

Challenges we ran into

  • Keeping every camera's calibration consistent, and deciding what to do when a camera has moved
  • Finding and removing the single point of failure we'd built in by hosting the app on a camera laptop
  • Turning "fail gracefully" into hard rules we could test by unplugging things

Accomplishments that we're proud of

  • Live fused skeletons from multiple cameras, rendered through a wall in VR
  • You can unplug a camera laptop mid-demo and the view stays correct

What we learned

Graceful failure is a design rule, not a feature you add at the end. "Less complete, never wrong" shaped every decision, from calibration to network layout.

What's next for Spectre

  • Autonomous scouts: run camera workers on recon drones and ground robots, located by their own onboard tracking instead of wall markers. A drone that loses signal drops out like an unplugged camera.
  • Real walls: in our demo the wall is a blanket. Next is adding sensors that pass through walls, like mmWave radar or Wi-Fi sensing. The pipeline doesn't care what the sensor is.
  • Backup server: a standby, so the fusion server stops being the last single point of failure.
  • Show the gaps: show each camera's coverage in the headset, so a blind spot looks blind, not empty.
  • Faster recalibration: trigger a check the moment a camera's markers drift, instead of waiting for the next minute.
Share this project:

Updates

Submission history