OffSight

An AI referee that shows its work: real tracking data drives the geometry, multi-agent reasoning makes the call like a VAR analyst, and a built-in guard blocks unsupported overrules.

Comment

OffSight

OffSight is an explainable AI football referee that reconstructs match moments from real tracking data, computes offside and contact geometry with deterministic precision, and reasons through the call the way a VAR analyst actually does, with a built-in guard that catches its own reasoning when it tries to overrule the facts.

Inspiration

FIFA's Semi-Automated Offside Technology tracks 150 million data points per match using a $100M+ camera rig most leagues on Earth will never afford. Meanwhile, the actual hardest part of officiating, not "where was the player" but "was this a violation," is still explicitly left to humans, because that judgment call is subjective, and no automated system fully owns it yet. We wanted to know: could an LLM reasoning pipeline take on that second, harder problem, transparently and honestly, without pretending false confidence on the close calls? OffSight is our answer.

What It Does

OffSight takes a moment from a football match, either a precomputed case or a freshly uploaded video clip, and returns a referee-grade verdict:

  • A deterministic geometric analysis of the offside line, contact proximity, and player velocities, computed to sub-metre precision
  • A five-agent reasoning chain that extracts the facts, retrieves the applicable law, makes the call, challenges its own verdict, and reconciles the two
  • A consistency guard that mathematically cross-checks every challenge objection against the underlying geometry, and overrides the AI's own reasoning if it tries to reverse a call using an unsupported claim
  • An end-to-end video pipeline, powered by real computer-vision tracking, so users aren't limited to pre-labeled footage; upload a clip and get a verdict

Supported situations: offside, foul/contact, clearance calls. Powered by real SoccerNet Game State Reconstruction match data, with a live GPU-backed inference path for arbitrary uploaded video.

How We Built It

Deterministic Geometry Engine

Before any model reasons about anything, we compute the objective facts in pure, non-AI code. Given the pitch-normalized position of every player and the ball at the frame the ball was played, we derive:

  • Offside line reconstruction: the second-last defender's position, projected across the pitch, compared against every attacker's position to compute a signed distance beyond the line, in metres. For an attacking team moving in the positive-x direction, an attacker at position x_a is offside relative to a defensive line at x_d when:

$$\Delta_{\text{offside}} = x_a - x_d > 0$$

with the offside margin reported directly to the reasoning pipeline in metres, giving every downstream agent a precise, auditable number instead of a binary flag.

  • Contact/proximity detection: pairwise Euclidean distance between all players per frame, flagging close-contact pairs for foul analysis
  • Velocity estimation: windowed positional deltas per player, used to establish whether a player was actively engaged in play at the moment of the call

This layer never touches an LLM. It is deterministic, unit-tested, and outlier-robust. Real tracking data is noisy (we found calibration extremes up to 85m against a normal operating range of -18 to 55m), so this layer rejects statistical outliers before anything downstream trusts the numbers.

Multi-Agent Reasoning Pipeline

Rather than a single prompt asking "was this offside," we built five specialized, sequentially-chained Claude-based agents:

  1. Fact-Extraction Agent: converts raw geometric output into a clean, minimal natural-language summary of what objectively happened, with no rule judgment yet.
  2. Rule-Retrieval Agent: classifies the situation type and retrieves the specific IFAB Law of the Game that applies.
  3. Verdict Agent: reasons over the facts and the rule to produce an initial call, with an explicit, cited reasoning chain.
  4. Adversarial Challenge Agent: is explicitly prompted to argue the opposite verdict as strongly and honestly as it can, forcing the system to stress-test its own conclusion rather than accept the first answer.
  5. Confidence Synthesis Agent: reconciles the verdict and the challenge into a final call, with a calibrated confidence label. On genuinely borderline cases, it says so; it does not force false certainty.

The Consistency Guard

The most critical component, and the one that separates OffSight from a system that just trusts its own LLM output: a deterministic post-reasoning guard that treats the Challenge agent's objections as claims to be verified, not accepted on faith.

For every objection raised, the guard computes a grounding score:

$$\text{grounded}(o_i) = \mathbb{1}\left[\min_{v \in \text{cited}(o_i)} \max_{f \in \text{facts}} \left(1 - \frac{|v - f|}{\tau}\right) > 0\right]$$

where cited(o_i) is the set of numeric values the objection cites, facts is the set of real values computed by the geometry engine, and tau is a small tolerance. In plain terms: an objection only counts as grounded if every number it references is actually close to a real, computed value. If it cites a margin, distance, or measurement that does not exist anywhere in the facts, it is discounted rather than treated as valid grounds for reversal.

If the Synthesis agent's final verdict contradicts unambiguous, complete-coverage geometry (the deterministic engine has no doubt), the guard overrides the AI's own call and restores the geometrically correct verdict, logging the override, the ungrounded objection, and its reasoning for full auditability.

This isn't hypothetical. During our own testing, the Challenge agent invented a nonexistent "acceptable margin of error" to argue a real 0.415m offside was actually onside, a fabrication with no basis in the rules or the facts. The guard caught it, discounted the ungrounded objection, and restored the correct OFFSIDE verdict. That exact failure and correction is reproducible and visible in the app's reasoning trace.

Video-to-Verdict Pipeline

To go beyond pre-labeled data, we integrated SoccerNet's Game State Reconstruction baseline (YOLOv11m detection, PRTReID re-identification, NBJW camera calibration) to run our full pipeline on arbitrary uploaded footage. Given the CPU-bound cost of dual-keypoint calibration, we deploy this stage on a GPU-backed worker (Google Colab, tunneled via ngrok), so a user-submitted clip is processed with real inference, not a canned response, and returned through the same API contract as our precomputed cases.

Challenges

LLM self-critique without ground truth is dangerous. An adversarial "challenge" step sounds like good practice, stress-testing your own conclusion, but an ungrounded challenge agent will happily invent plausible-sounding technicalities to win the argument. We only discovered this by testing against real geometry we already knew the answer to. Without a deterministic ground truth to check against, this failure mode would have shipped silently.

CPU-bound computer vision is a real bottleneck. Dual-HRNet-style camera calibration is heavy, and without GPU acceleration a single short clip took 20+ minutes to process. Moving inference to a GPU-backed cloud worker was the only path to a usable upload experience.

Camera coverage isn't guaranteed. Real broadcast footage sometimes doesn't have the full defensive line in frame. Our geometry engine explicitly flags incomplete coverage rather than guessing, and the reasoning pipeline treats "insufficient data" as a valid, honest outcome.

What We Learned

Deterministic ground truth and LLM reasoning solve different problems, and conflating them is where systems fail. Geometry should never be estimated by a language model; judgment calls should never be made by unchecked arithmetic. The right architecture keeps these separate and uses one to audit the other. That is what actually makes an AI-driven officiating system trustworthy rather than just plausible-sounding.

What's Next

  • Expand rule coverage beyond offside/foul/clearance to handball and denial-of-goal-scoring-opportunity calls
  • Precedent retrieval: embed and search past ruled cases to surface similar historical calls as additional context
  • Real-time inference on live broadcast feeds, not just uploaded clips
  • Multilingual verdict explanations for global accessibility

Built With

Share this project:

Updates

Submission history