PackageCare

Damaged packages are annoying, dealing with it doesn't have to be.

Comment

Inspiration

There are few things worse than finally getting the package you've been waiting for, opening it up, and finding it damaged. The excitement turns into dread about the work it takes to sort it out: scrubbing through doorbell footage, guessing at timestamps, and writing everything up by hand. Ring already tells you a package arrived. We wanted to show how it arrived, without turning an uncertain model guess into an accusation.

What it does

  • Connects to Ring through the official Partner API: live view over WHEP, plus in-browser capture of a 12-second clip with the original bytes preserved and labeled as a browser recording.
  • Analyzes handling: a video classifier scores every 2-second window for toss/drop versus placement. The strongest segment becomes a timeline observation linked to the exact frames, with a handling-activity chart and a one-tap slow-motion replay. When the handling isn't visible, it says "Unable to assess" instead of guessing.
  • Puts the recipient in charge: they confirm, correct or dismiss each observation and record damage, notes, a shipment reference and photos. Model observations and recipient statements stay separate, because a toss doesn't imply damage and damage doesn't prove a toss.
  • Exports a report: it is previewed first, frozen as a version, then exported as a ZIP with a printable report, the evidence frames, the source video and a SHA-256 manifest. Nothing is sent anywhere automatically.

How we built it

  • App: React + Vite + TypeScript UI, an Express API, FFmpeg/Sharp media preparation, and strict domain validation (every observation must cite frames inside its own time range).
  • Ring: Partner API token connection, device discovery, WHEP live view, browser capture and recording-history import.
  • The handling model: real, licensed footage of packages being thrown is nearly impossible to get, and general multimodal models (Amazon Nova Lite, Qwen3-VL, Mistral Large 3 and a local Qwen 27B) either missed brief tosses or invented details. So we:
    1. generated ~490 labeled doorbell clips with an AI video model (Seedance via fal.ai), varying camera, setting, lighting, courier, parcel and action (toss, drop, placement, no handling, hidden);
    2. degraded them to look like real cameras (fisheye, low resolution, blur, noise, heavy compression), giving ~2,000 training clips;
    3. extracted features with a frozen Meta V-JEPA 2 video model and trained a small classifier on top;
    4. tested only on footage it had never seen: clips from a different generator (Kling) and real recordings.
  • AWS (us-east-2): one CDK stack. Browser uploads go straight to a private S3 bucket through presigned POSTs. Step Functions orchestrates retrieve → observe → verify on an arm64 Lambda container running FFmpeg, PyTorch (CPU), V-JEPA 2 and our classifier. DynamoDB holds deliveries, reviews and frozen reports, API Gateway + Lambda serve the app, and EventBridge triggers a daily retention cleanup.

Results (honest numbers)

  • Clips from another generator: 14 of 15 tosses and drops caught; 23/29 correct overall.
  • Real recordings never used in training: no false concerns on 6 real clips without rough handling; caught the clear real toss, missed a distant CCTV throw.
  • Weakness: 3 of 10 careful placements were flagged. That is exactly why the recipient has the final say, and why scores are shown as evidence ("strong / moderate signal"), never as a verdict.

Challenges we ran into

  • No real handling footage. The Ring Playground streams a parcel already at rest, and we don't own a Ring camera. Synthetic training data plus held-out testing from a different generator was our way through. Every AI-generated clip is labeled in the app, the report and the video.
  • General vision-language models weren't reliable for this narrow task. They abstained on obvious tosses or described things that weren't there. A small task-specific classifier on strong frozen video features worked far better.
  • CPU video inference inside Lambda. New AWS accounts cap Lambda memory at 3 GB (~2 vCPU), our first cloud run timed out from thread oversubscription, and cold starts on a 4 GB image needed longer FFmpeg timeouts. After tuning, a clip is analyzed in ~1.5 minutes warm, for about a cent.
  • Restraint by design. Making "Unable to assess" a first-class, well-presented outcome took as much work as detection.

Accomplishments we're proud of

  • An end-to-end workflow from a live Ring stream to an exportable, versioned, hash-manifested evidence report.
  • A handling classifier trained without a single real training clip, evaluated honestly on held-out footage.
  • The full serverless pipeline deployed on AWS and processing real uploads.

What we learned

"Unable to assess" builds more trust than a confident wrong answer. Synthetic data is a practical bootstrap for rare events, as long as you evaluate on data from somewhere else. And infrastructure limits (memory caps, cold starts, CPU threads) shape ML product design as much as model accuracy does.

What's next

More real, consented doorbell footage for evaluation; per-camera calibration; and, with an authorized carrier integration, a recipient-approved hand-off of the reviewed report to the carrier's claims team (shown in the demo as a labeled concept only). PackageCare will never identify drivers or send anything without the recipient's say-so.

Built With

Share this project:

Updates

Submission history