Inspiration
RaidRift started with one frustrating moment that every IRL streamer knows: a raid.
A raid is the biggest gift a Twitch community can give. Hundreds of people arrive at once, ready to take part. On an IRL stream, though, almost nothing happens. A banner flashes, chat explodes for ten seconds, and the streamer keeps walking down the same street. The raiders arrived, but they never really got there.
That became the spark: what if a raid physically showed up in the streamer's world?
During research I couldn't find anything that did this. Raid alerts are 2D overlays pinned to the corner of the screen. AR filters decorate the streamer's face but don't give viewers anything to do. Chat games mostly live on a PC next to OBS, completely disconnected from someone walking outside with a phone. None of them turned a raid into people you can see standing on the sidewalk.
So I built RaidRift: raid real life.
What it does
When a Twitch raid hits, an AR portal tears open in the streamer's real surroundings. Raiders type !join and step out of it as their own heroes: a Knight, a Barbarian, a Mage or a Rogue, standing on the actual ground in front of the camera.
Then chat takes over. Viewers simply tell their hero what to do: "walk right then jump", "come closer", "go sit on that chair". The AI listens and their character moves.
RaidRift also sees the world. On-device object recognition spots real things around the streamer, like chairs, benches and tables, so raiders can send their heroes to them and interact with them. The real world stops being a backdrop and becomes the level.
Every raider controls only their own hero, so a raid becomes a crowd of real people's avatars walking alongside the streamer.
Next comes the Rift Trail, a quest that happens while the streamer walks. Crystals appear on real surfaces around them. Collect three and a shrine rises; restore it and the whole raid celebrates together, live on stream.
A raid stops being a notification and becomes a party that shows up.
How I built it
RaidRift is an AR app built in Unity, developed with Codex.
- ARCore scans the real ground and measures depth. The portal only lands once the surface is stable, and it stays anchored where it landed.
- Object recognition runs on the phone and detects real objects in the camera feed. Each detection is placed in 3D space, so heroes can walk to a real chair and use it.
- The heroes respect reality. They walk only on ground the phone has actually measured, and real-time pathfinding routes them around obstacles. If there's no floor, they don't step.
- Chat → AI → motion. Raiders prompt their hero in plain language, in English or Ukrainian, and it turns into movement, jumps, emotes and object interactions instantly.
- Built for broadcasting. The phone composites the AR scene and game sound and streams it straight to OBS over SRT/RTMP, without the streamer's controls showing up on stream.
- Twitch backend: raid and chat events, viewer identity, quests, leaderboards and a Shards economy, with rewards controlled by the server so nobody can cheat them from a phone.
I deliberately made the streamer's phone the stage. IRL streamers don't carry a PC, so everything happens in one app, right where the stream is happening.
Challenges I faced
- Meshes were the hardest part. Getting a phone to understand the real world well enough for characters to stand on it was brutal. Scanned surfaces came out patchy, uneven and full of holes, and early heroes floated in mid-air, sank into the floor or walked straight off the edge into nothing. I had to teach the characters to trust only ground the phone had actually measured: depth can block a hero but never invent a floor.
- Finding the right tech. I switched stacks more than once before settling on Unity. Each attempt solved one problem and broke another: anchoring, character animation, rendering performance, streaming output. Unity was where AR, animated characters and real-time rendering finally came together in one place.
- Turning detections into places. Recognizing a chair in a 2D camera frame is one thing. Knowing where it actually stands in 3D, so a hero can walk over and sit on it, is much harder.
- The streamer never stops walking. The party had to follow naturally, without teleporting or piling into one spot.
- One viewer, one hero. In a raid of hundreds, nobody can hijack someone else's character.
- One phone does everything: AR tracking, depth, object recognition, AI commands, rendering, pathfinding and live encoding, all at once.
What I learned
Grounding beats spectacle. The moment that felt magical wasn't the effects. It was a character stopping at a real curb because the phone knew the ground ended there. I also learned that viewer interactivity needs strict ownership from day one. A raid only feels personal if your hero is yours.
What's next
- A live raid between two real Twitch channels, end to end.
- Smarter object interactions and bigger quests that react to where the streamer actually is.
- Integration with IRL streaming apps like IRL Pro, so any IRL streamer can let their raids in.
Raids shouldn't just be announced. They should arrive.
Built With
- codex
- unity
