Touchline AI

• 

  The annotated match video with the pitch template locked on, the top-down view with player trails, team metrics, and the agent timeline

• 

  The agent’s segment-2 decision expanded: it saw camera drift, so it chose to update the calibration.

• 

  Before and after on the same frame: misaligned lines (0.40) vs locked-on (0.90). Most convincing at a glance.

• 

  Every quality signal across all four segments, initial → corrected, with thresholds and failures in red.

• 

  The four-point calibration on the real frame, with the whole pitch projected back.

• 

  The occupancy heatmap with its plain-English definition.

Inspiration

College teams film games on a phone from the stands. I wanted to turn that footage into tactics like team shape and heatmaps. But in my test clip the camera drifted about 135 pixels, so every player position silently slid several metres. Numbers that look right but aren't are worse than no numbers. So Touchline AI became analysis that knows when it's wrong.

What it does

Upload a 10–30 s clip and mark four points on the pitch. An agent then checks each ~4 s segment:

• Perceive: track players, map them onto a top-down pitch, and measure its own quality.
• Decide: camera moved → update the calibration; tracking broke up → sample frames more densely.
• Act: rerun that segment and keep the fix only if it measurably helps on the same frames.
• Escalate: after two failed attempts, flag the segment and ask the coach.

It's rule-based, with no LLM in the loop, so every decision is reproducible and logged.

How I built it

OpenCV 5 does the vision: field segmentation, line detection, homography calibration, camera-motion tracking (ORB / optical flow + MAGSAC), Kalman tracking, and a YOLOX-S detector run through cv2.dnn. To check the geometry, it warps each frame's painted lines onto the calibration frame and scores how well they line up.

Stack: FastAPI, SQLite, React/TypeScript, Docker. On AWS, an EC2 worker processes clips from encrypted S3 storage.

Results

• Fixed camera drift in all 4 segments of a real clip (line alignment 0.2–0.5 → 0.8–0.9)
• Position error after camera movement: 7.9 m → 0.6 m
• Right action in 8 of 9 stress tests
• Runs end to end on EC2 + S3

Challenges

• My quality check fooled me: photographers at the pitch edge slipped past the spectator filter, so I fixed the filter instead of loosening the threshold.
• Checks that fought each other: a valid camera fix was vetoed by unrelated tracking noise, so each fix is now judged on its own evidence.
• Lying video files: a header claimed 120 fps, so I now read timing from the decoded frames.
• Real-world AWS: a region lock, free-tier limits, and a changing IP, each solved with a script.

What I learned

• "I'm not sure" is a feature, not a failure.
• Compare fixes on identical frames.
• Say what you don't measure.
• Test your tests.

What's next

Catching calibrations that are wrong from the start, adding player re-identification, and getting human-verified evaluation labels.

Built With

• amazon-ec2
• amazon-linux
• amazon-web-services
• aws-iam
• aws-systems-manager
• boto3
• docker
• docker-compose
• fastapi
• github
• numpy
• onnx
• opencv
• pydantic
• pytest
• python
• react
• scipy
• sqlite
• typescript
• uvicorn
• vite
• yolox