Inspiration

Every friend group has the argument: whose room is better and where to meet up? We wanted a way to settle it live, face to face, without anyone downloading a new app. FaceTime is already where we hang out, so we put the game inside it, with an AI as the referee nobody can argue with.

We were inspired by how Krillion created feeling of picking something niche and getting rewarded for it. We were also inspired by Omoggle, a 1v1 competition judged by AI.

What it does

Envoggle is a live 1v1 scavenger hunt on FaceTime.

1. In an iMessage chat, open + → Envoggle and tap Start Game. Your friend gets an invite, and the Envoggle referee calls you both on one Group FaceTime.
2. The referee asks each player to say their name, then announces a category, such as "Rare things that are blue", "Most calories" or "Longest thing". There are 15 categories, and each game draws 3 of them.
3. Both players have 20 seconds to find something in their room and hold it up to the camera.
4. The referee finds every object in each player's frame that fits the category, picks the best one, and scores it out of 1000. More niche / more fitting to the category means more points. Then it reveals the winner of the round on its own video tile.
5. After 3 rounds, it shows the final scores.

How we built it

• iMessage extension (Swift and SwiftUI): saves the two FaceTime numbers for each conversation, sends the invite bubble, and asks our server to start the call.
• Server (TypeScript and Node.js): runs the game state and asks Chert to place one Group FaceTime call to both players.
• Chert FaceTime line: bridges the FaceTime call into a LiveKit room.
• Referee (LiveKit Agents): joins the room and watches the call video. Chert gives a group call as a single FaceTime window feed, so the referee works out where each player's tile is in that window and crops each player out of it. It renders its own animated tile (countdown, scan, scores) and publishes it as video.
• Scoring: a vision model (Gemini 3.5 Flash in our demo; Claude is also supported) returns every object in the frame that fits the category, with a bounding box and a rating. We turn the best one into a non-round score out of 1000.
• Voice: speech-to-text listens for the players' names only and ignores ordinary words using a dictionary. The referee speaks with Piper, a local text-to-speech model, so it costs nothing and has no rate limits.
• Ops: everything runs on a small Linux VPS as systemd services, reachable only over Tailscale, so the game keeps running with our laptops (where the server was originally hosted) closed. The referee can record every call frame-perfectly with ffmpeg, which is how we made our demo video.

Challenges we ran into

• One feed for two players. A Chert group call gives one window feed and one mixed audio track, not one per player. We had to find each player's tile inside the FaceTime window, cope with layout changes, and notice when only one person was still on screen.
• Scoring that felt fair. Our first version rated the whole frame, so a messy desk beat one good object. Making the model list every eligible object first and then rate each one fixed it. Round numbers like 500 felt fake, so we made scores non-round.
• Names, not noise. The referee kept hearing words like "good" or "Envoggle" as player names, so it now accepts only words that look like real names.

Accomplishments that we're proud of

• A full game happens inside apps everyone already has: Messages and FaceTime. Nobody installs anything to play.
• A real AI referee on a live group call: it listens, talks, shows its own video tile, and judges in a few seconds.

What we learned

• Real-time video is mostly about timing: frame pacing, event-loop stalls and audio gaps matter more than the model.
• With vision models, asking them to list the eligible objects and then rate each one works far better than asking for a single overall rating.
• Small, local models (TTS, background removal) are often the right call next to a big cloud model.

What's next for Envoggle

• More players, team modes, and categories made up from the call itself.
• Leaderboards and highlight clips cut automatically from the recordings.

Built with

TypeScript, Node.js, Swift, SwiftUI, iMessage, iOS, FaceTime, Chert, LiveKit, LiveKit Agents, Gemini, Claude, OpenAI, Piper, ffmpeg, Transformers.js, BEN2, ONNX, sharp, Tailscale, Linux, systemd, Xcode

Did you implement a generative AI model or API in your hack this weekend?

Yes, in three places:

• Judging (Gemini 3.5 Flash; Claude is also supported): a vision model looks at each player's frame, lists every object that fits the round's category with a bounding box, and rates how rare or extreme each one is. We turn the best rating into a score out of 1000 and use the box to outline the object on screen. We used a generative vision model because the categories are open-ended (rare blue things, the most calories, the longest thing), and no fixed classifier could judge "niche" across whatever happens to be in someone's room.
• Listening (OpenAI speech-to-text): transcribes the call so the referee can hear each player say their name. We filter the transcript so only real names count.
• Building: we used AI coding assistants (Claude Code and Codex) while building and debugging.

Built With

• chert