-
-
Drop in a slo-mo replay and RoboRef tracks the players and ball, then makes the call with all its reasoning on screen.
-
Real World Cup footage judged by RoboRef, with annotated clips, the stats behind every call, and fan voting.
Inspiration
Every match day, my friends and half of social media are locked in the same argument: that was a robbery, the ref rigged it, and that was never a foul. Watching yet another World Cup debate spiral, I wondered: why are the most contested decisions in sports still made by a single human under pressure? An AI, if trained unbiasedly, is emotionless. It doesn't hear 80,000 fans screaming, doesn't know which team is the favorite, and doesn't carry a reputation to protect. It runs on statistics and data, so it can function the way we wish referees could: objectively and the same way every single time. RoboRef is my first step toward that — pushing for unbiased refereeing. Additionally, I have family who enjoy watching the games but fail to understand what truly goes into the decisions being made. RoboRef also helps those who are confused with open-ended explanations and reasoning for decisions.
What it does
RoboRef adjudicates soccer tackles from slow-motion replay footage — exactly the footage a human VAR reviews. You drop in a clip, tell it the two jersey colors, and it returns a verdict (FOUL or PLAY ON, penalty vs. free kick), an annotated video with the attacker, defender, and ball tracked frame-by-frame, a telemetry sheet of every metric behind the call, and a formal two-sentence referee ruling grounded in Law 12 of the Laws of the Game. Nothing is a black box: the exact overlap percentage, ball-possession evidence, and who-played-the-ball-first analysis are shown to the user — open reasoning, so fans can argue with the data instead of the ref.
How we built it
A three-tier pipeline: The Watcher (computer vision) runs YOLOv8 + ByteTrack to track every player and the ball, with several layers we had to engineer on top — re-stitching player tracks that break exactly at the tackle (occlusion kills IDs at the worst moment), a speed-gated ball trajectory that rejects false-positive detections; jersey-color matching in perceptual Lab color space to identify attacker vs defender, and camera-pan compensation from background optical flow. The Thinker (heuristics engine) converts tracks into a verdict: contact only counts near the ball, possession requires the attacker to be measurably closer to it than the defender, and a clean tackle is detected by who emerges with the ball. The Speaker uses the Gemini API with FIFA's Law 12 injected into the prompt to write the official ruling. FastAPI backend, React + Tailwind frontend, and a full evaluation harness that scores the system against human-labeled clips.
Challenges we ran into
Data is EXTREMELY hard to find. There is no public dataset of labeled foul/no-foul clips with usable camera angles — I had to dig for footage myself, clip by clip, through YouTube, and hand-label every verdict. That scarcity reshaped the whole project: with only a handful of labeled incidents, training or fine-tuning a model end-to-end was off the table, so the pipeline had to become less complex — pretrained detection models plus transparent, physics-based decision rules tuned against a small evaluation set, instead of a learned black box. Ironically, that constraint became the product's identity: every verdict is explainable. Beyond data, live wide-angle footage proved too noisy to trust (we scoped to slo-mo close-ups), and two "obvious" upgrades — slowing footage via frame interpolation and pixel-level segmentation masks. However, both measurably hurt accuracy when we A/B tested them, so we moved on without them.
Accomplishments that we're proud of
- 5/5 correct verdicts on our human-labeled set of real broadcast tackles — including correctly ruling clean tackles as play-on, not just calling everything a foul.
- Camera-proof metrics by design: every measurement is in body-heights and every time window scales with frame rate, so verdicts hold up regardless of zoom level or replay speed — validated on footage from five different broadcasts.
- An honest engineering culture in miniature: every idea was measured against labeled data, and the failed experiments are documented with numbers instead of quietly deleted.
- A complete, demoable product — upload to verdict in seconds, with the reasoning on screen.
What we learned
The decisive signal in refereeing isn't bodies colliding — it's the ball. Overlapping players mean nothing; who had the ball, and who played it first, decides everything, and our accuracy jumped when the pipeline internalized that. We also learned that tracking reliability beats model sophistication (a fancier segmentation model with worse tracking lost to simple boxes). And with scarce data, transparent rules beat trained models: when RoboRef gets a call wrong, we can open the telemetry, see exactly which measurement misfired — a contact threshold, a possession distance — and fix that one rule, then instantly re-verify every past verdict. A trained model that gets a call wrong offers no such handle: you can't ask it why, and with only a handful of clips, you can't retrain it into better behavior either.
What's next for RoboRef
- A real data pipeline — automated collection, clipping, and labeling of broadcast incidents so the evaluation set grows from a handful of clips to thousands, unlocking learned components where they beat the heuristics.
- More types of calls — handballs, offsides, dives/simulation, and card-severity recommendations, each with the same open-reasoning treatment.
- 3D perception — today's system reasons in 2D pixels; the next leap is true spatial understanding, either AI-based (multi-camera or monocular depth estimation) or sensor-based with LiDAR, so contact, ball position, and player separation are measured in meters on the pitch instead of pixels on a screen.
Log in or sign up for Devpost to join the conversation.