System Architecture

R.E.A.C.T, Real Environment Augmented Combative Tactic

Inspiration

My friend was a bit too annoying yesterday, so I wanted to build a game where I could ~~shoot~~ him for fun, without the consequences of course.

What it does

R.E.A.C.T is a real world team shooter played with phones and a laptop.

Scan the arena. Walk the building with an iPhone and our scanning app. It records camera frames, ARKit poses and LiDAR points, uploads them to the laptop, and trains them into a photorealistic 3D Gaussian splat map. The map sharpens live in the viewer while it trains.
Suit up. Each player wears a phone on their chest running our R.E.A.C.T Player app, along with a printed AprilTag bib that identifies them.
Play. Point a finger gun at an opponent's tag to shoot. Three hits take a player down, and you clap to reload. A downed player is out until they walk back into their own team's base, and they cannot shoot or reload on the way.
Win. Matches are timed. When the clock runs out, the team with the most kills wins.
Spectate. The laptop shows the scanned building in 3D with every player moving live as an avatar in their team's color. It also shows health, shot tracers, a kill feed, the score and the match clock. You can fly around the map or watch through any player's eyes. Phones announce what matters by voice, such as "You're down. Back to your base."

How we built it

Scanning and maps. A native iOS app (Swift, ARKit, LiDAR) captures the dataset. A Node and TypeScript server trains it into a Gaussian splat with Brush on the Mac's GPU (Metal). The viewer renders it in the browser with Three.js and Spark, and it also imports Scaniverse scans.
Position tracking. The R.E.A.C.T Player app streams ARKit camera poses at 30 Hz, along with step detections, a gyro heading and the compass, to a FastAPI game server. For each player, a particle filter fuses those signals and is constrained by the walls of the scan. When the camera is blocked or loses tracking, step counting takes over. When the camera comes back, the gyro realigns it. The host places each player once on the top down map, and the server follows them from there.
Shooting. Phones stream their camera frames to an aiohttp shooting server. It runs MediaPipe hand landmarks to recognize a one handed finger gun pose, then uses the pupil apriltags library (with OpenCV) to find which bib the finger points at. Claps reload, and they are detected by the microphone or the camera. When several phones hear the same clap, the loudest one owns it.
The match. The shooting server follows everyone's position from the game server and runs the rules, including health, kills, respawning at bases and the match clock.
Testing without people. A simulator walks virtual red and blue players around the map, and a fake scan generator stands in for the iPhone scan, so we could test the whole stack on one laptop.

Challenges we ran into

Indoor tracking is hard. GPS does not work inside, and ARKit drifts or freezes when someone covers the camera. We built a fusion layer that detects when the camera and the step counter disagree and decides which one to trust.
Who got shot? We first tried recognizing players by their clothes from a quick body scan before the game. Similar outfits and changing lighting made that unreliable, so we switched to printed AprilTag bibs, which are fast, cheap and almost never misread.
Lining up coordinate systems. The scan, the 3D viewer, the 2D map and each phone's ARKit session all have their own coordinate frame, and they all had to agree on where the floor and the bases are.
Real world networking. Phones lock their screens, walk out of WiFi range and drop connections mid match. One stuck phone once slowed position updates for everyone to one per second. We gave every connection its own send queue and made phones reconnect automatically.
Browsers and HTTPS. Phones only give web pages camera and motion sensor access over HTTPS, so the servers generate their own certificates on the local network.

Accomplishments that we're proud of

Going from walking through a room to playing in a photorealistic 3D copy of it, with no special hardware beyond phones and a laptop.
Live spectating, where you can watch real people move through a Gaussian splat replica of the building as they play.
A complete game loop running end to end, covering tracking, shooting, health, respawning and the match clock.
An automated test suite covering the tracker, the game rules and the shooting pipeline on real camera images.

What we learned

How Gaussian splatting works, and how to train splats quickly enough to use them during a hackathon.
Sensor fusion and particle filters for indoor positioning, and how often each sensor lies.
How much a simple fiducial marker like an AprilTag beats a clever machine learning approach when the result has to be right every time.
That simpler rules often make a better game.

What's next

Automatic player placement, using ARKit relocalization to find each phone's position instead of placing players by hand.
Detecting walls automatically from the scan.
Joining a match by scanning a QR code in the app.
More game modes, such as king of the hill, and a native Android player app.