TPlus Vision Enrichment

Don't post the still if it doesn't show the launch event — OpenCV 5 + AWS gate alert images for TPlus webcast milestones.

Comment

Inspiration

I run TPlus, a small launch-alert tool for friends who care about spaceflight but cannot watch multi-hour webcasts. Timeline + OCR already decide when a milestone fires (liftoff, MECO, stage sep, landing…). The painful failure mode was the still image on the alert: black frames, HUD cards, wrong phase, or fireball mush that does not actually show the event. A misleading thumbnail is worse than no image.

That mapped cleanly to a computer-vision question for the OpenCV × AWS contest: does this frame show named event X? If not, do not post it.

What it does

Around each milestone, TPlus extracts a short window of frames (not the full VOD). OpenCV 5 scores those frames. We treat the product gate as binary:

Does this image show [liftoff / MECO / stage sep / …]? → yes / no

If the answer is not a confident yes, the alert stays text-only. Schedule + OCR remain authoritative for timing; vision only decides whether a still is allowed.

Frames, human picks, and score artifacts live in Amazon S3. The same batch_score.py path runs locally or on EC2 (OpenCV 5.0.0 on a short-lived t3.micro; terminate-after).

How we built it

  • Window extract: yt-dlp -g + ffmpeg around milestone times
  • Gold: human best-still picks + yes/no shows-event labels (shows_event.jsonl)
  • OpenCV 5 heuristics: HSV plume / coast, sharpness, mid-contrast, HUD and low-value penalties → label + rank (score_frame.py, batch_score.py)
  • AWS: S3 pack store (tplus-vision-scenicmind, us-east-2) + EC2 batch; summaries record OpenCV version, host, and timing
  • Eval: precision on yes first (false posts hurt); high threshold prefers refusing a still over attaching a wrong one

Code and writeup are in the Cue/TPlus monorepo under apps/tplus/src/webcast/vision/

Challenges we ran into

  1. "Pretty still" ≠ "shows the event." Ranking for aesthetics over-fit fireballs and under-served coast / MECO. Reframing labels as yes/no shows event X matched the product pain.
  2. Webcast noise: logos, replay cards, and telemetry HUDs look "informative" but fail the post test — they must be hard negatives.
  3. AWS account friction: a managed Free Tier org SCP blocked some EC2 shapes; we still landed a real OpenCV 5 batch on t3.micro + S3 and terminated the instance.
  4. Honest metrics: top-k vs human picks is still weak; we report that and optimize precision on yes for the gate instead of claiming solved detection.

Accomplishments that we're proud of

  • Reframed the problem as a product gate, not a leaderboard chase: “Does this frame show event X?” — so TPlus can refuse misleading alert stills.
  • Shipped a real OpenCV 5 path (score_frame / batch_score) on authentic webcast windows, not toy images.
  • Ran the same scorer on AWS: frames in S3, batch on EC2 with OpenCV 5.0.0 (~420 frames / ~4s per flight), then terminated the instance.
  • Built human gold that matches the pain: best-still picks plus yes/no shows-event labels, with HUD / wrong-phase frames as hard negatives.
  • Kept the system honest: timeline + OCR still own alert timing; vision only decides whether an image may post — and we report weak top-k instead of papering over it.

What we learned

  • Product-shaped labels beat generic scene taxonomies when the user harm is specific.
  • Soft enrichment (text alert always OK; image optional) is the right first ship for sparse ops tools.
  • Reproducible cloud proof for this contest is boring on purpose: same scorer, S3 in/out, short EC2 run, kill the VM.

What's next for TPlus Vision Enrichment

Grow yes/no gold across more flights, raise gate precision, then optionally reuse the same evidence for "schedule lag" overrides (e.g. visual MECO before a late clock) — without letting vision fire alerts alone.

Built With

Share this project:

Updates

Submission history