Seen Off

Carrier-pickup proof for people who ship from home, built on the Ring Partner API.

Comment

Inspiration

A home seller on Poshmark, eBay, Etsy or Mercari ships three to thirty parcels a week and leaves them on the porch for a scheduled USPS or UPS pickup. Many already treat the Ring doorbell as their receipt. @kancewe on X: "I put it on a table in front of my Ring doorbell, making sure the Ring can see the label. This gives me a record of the physical pickup." @AQualityMess: "I literally have ring camera footage of the mailman taking them and they never arrived at the fucking post office according to the tracking numbers. This is twice in a month this has happened to me." @Violetsvanity7 watched the carrier take one of three parcels and leave.

Every Ring package entry we found points the camera at parcels arriving. Seen Off points it at parcels leaving.

What it does

Seen Off gives one verdict per pickup day, with the Ring frames behind it:

  • taken: every parcel left the porch and every label shows a carrier scan.
  • missed: parcels are still on the porch when the window closes. The seller gets an alert with the parcel count while there is time to drop them off.
  • part-taken: "1 of 3 parcels taken, 2 still on the porch."
  • unscanned: the parcels left but a label shows no carrier scan 24 h later. The seller gets a dispute kit: one HTML file with the window-open and window-close frames embedded beside their sha256, every timestamp, each parcel's label and tracking state and the full event log, plus a carrier claim drafted by Qwen3-VL on Bedrock.

When a frame cannot be read, the seller is asked to check the Ring app. Seen Off never reports a pickup it did not see.

Results:

  • 29 of 30 pickup days called correctly (taken 10/10, missed 10/10, part-taken 4/5, unscanned 5/5).
  • A parcel on the porch is read correctly in 59 of 60 frames (98.3%), 35 of 35 on real fixed-camera footage, parcel count exact on 49 of 53, median 1.33 s per frame.
  • 0 of 15 days with parcels still on the porch were reported as taken.
  • Every empty porch is called empty: 30 of 30.
  • Two must-fail rows show the checks can go red: a vision stage that always answers "present" scores 30/60 on frames and 10/30 on days, and a verdict logic mutated so a missed day reads as taken scores 22/30.
  • Webhook receiver: acknowledgement p95 1 ms over 100 signed deliveries, 100 of 100 duplicates acknowledged without reprocessing, 10 of 10 forged bodies answered 401.
  • Missed-pickup alert median 1.5 s after window close.
  • Across 60 recorded pickup days (the 30-day set plus a second 30-day set built with a different seed), no day with parcels still on the porch was reported as taken.
  • Every drafted claim is checked before it is shown: a time, tracking number or parcel count that is not in the packet gets the draft refused. Live on Bedrock, the unscanned demo day's claim passes on the first draft and cites the 3:08 PM handoff.

The seller page (npm run serve) arms a pickup day (doorbell, window, one label per parcel), fills a two-lane timeline live over server-sent events (on the porch, with the carrier) and opens the evidence packet. Each verdict has its own colour and glyph, the window-open and window-close frames sit inside the verdict, and every time reads in the seller's zone ("3 parcels still on the porch at 5:00 PM"). The demo plays on page load at https://seen-off-demo.fly.dev: a missed day, a part-taken day and a day where everything leaves the porch and the carrier never scans it, back to back, each step carrying the seller post on X it answers. A Replay control plays any of 30 recorded pickup days through the production receiver, runner, ledger and vision stage, with no credentials: each frame gets the reading Qwen3-VL gave for those exact bytes, looked up by sha256, or a live Bedrock read when a key is set.

How we built it

Ring webhook (motion_detected / button_press)
  -> verify X-Signature, record event, ack            src/server.ts, src/ring/webhook.ts
  -> settle delay, then a snapshot (WHEP as fallback)  src/pickup/runner.ts
  -> vision stage: parcels present, how many           src/vision (Qwen3-VL on Amazon Bedrock)
  -> ledger (SQLite): baseline, motion, close counts   src/store/ledger.ts
  -> window-close verdict, 24 h tracking verdict       src/pickup/verdict.ts
  -> alert + evidence packet                           GET /days/:id

Ring surfaces called, each in its own module: OAuth refresh-token grant and the Playground token (src/ring/client.ts); GET /v1/devices?include=status,capabilities,configurations; POST /v1/devices/{id}/media/image/download with the 303 to the signed URL and retries on RECORDING_NOT_READY; GET /v1/history/devices/{id}/events?event_types= with links.next paging; WHEP live view through a Python aiortc worker that always closes the session (worker/whep_frame.py); v1.1 webhooks with X-Signature HMAC-SHA256 over the raw bytes, a 5 s acknowledgement budget and meta.request_id idempotency (src/ring/webhook.ts, src/ring/signature.ts); and account linking, both the Ring-driven one-way link with HMAC nonce (default) and partner-initiated OAuth with PKCE (src/ring/oneway.ts, src/ring/oauth.ts). npm run ring:setup prints the portal values; npm run ring:preflight times auth, devices, history, snapshot and WHEP.

The verdict reads the whole window, not two frames. A departure needs positive evidence, a count below the baseline, because a wrong "taken" silences the alert. A close count above the baseline with no frame ever showing fewer parcels is a missed pickup; with a frame that showed them leave, it is a contradiction, and Seen Off claims nothing. Every scheduled check lives in the ledger, so a restart mid-window still delivers the verdict and the alert.

Shippo supplies labels, tracking and pickup rescheduling (src/shippo/client.ts). The dispute kit is GET /days/:id/export (src/pickup/export.ts); the claim is POST /days/:id/claim, a second Qwen3-VL call on the same Bedrock endpoint, with a grounding check that refuses any time, tracking number or count the packet does not hold (src/pickup/claim.ts). The public demo runs replay-serve --public on Fly.io: the internet-facing app is read-only (arm, delete and Ring routes answer 403) while the replay drives a loopback-only twin over the same ledger. The repo has 120 tests and a mutation gate of 55 mutations, each of which must turn the suite red.

Challenges

  • A parcel is small, grey and sometimes under infrared. Seven of the 60 frames are night or infrared, measured from pixels, and they score 6 of 7. The verdict logic is built around that: it needs a count below the baseline to call a pickup, and re-reads an empty porch at window open before accepting it.
  • Webhooks arrive at least once. The first version of idempotency marked a delivery before handling it; a failed first attempt was then swallowed on Ring's retry. Seen Off marks after success, tested with 50 concurrent copies running the handler once.
  • Linking an account. Two full flows were built because the documented partner-initiated OAuth is invitation-only (friction entry 5).
  • Sandbox shape. The Playground produces live-view sessions, not webhooks, so the receiver is driven by HMAC-signed v1.1 deliveries (friction entry 1).

Accomplishments

  • The only entry we found aimed at outbound pickups, measured on the decision that matters, the outcome per pickup day, not only per-frame counts.
  • A vision stage that reads a real doorstep clip exactly, 35 of 35 frames, and never raises a false alarm on an empty porch.
  • A receiver that follows the Ring delivery contract requirement by requirement, with the code and the test behind each in docs/ring-contract.md.
  • An open-source fix to Ring's own sample app for the retry path (Open Source mini challenge below).
  • A judge can open one URL and watch three pickup days land, then download the evidence kit and have Bedrock draft the carrier claim from it, with no account and no key.
  • A claim writer that cannot put an invented time or tracking number in front of a carrier: the grounding check is a must-fail test, red when the time check is removed.

What we learned

  • Put the model behind a one-method interface and the provider becomes a config choice: the same 60 frames ran on Nova Lite (58 of 60) and on Qwen3-VL on Bedrock (59 of 60) with no change outside the provider.
  • A false "taken" is worse than a false "missed", so the logic is asymmetric on purpose.
  • The Ring contract rewards treating a delivery as at-least-once from the first line.

What's next

  • Link a seller's own Ring account through the Appstore flow and run the same verdicts on their doorbell.
  • Carrier pickup rescheduling from the missed-pickup alert (Shippo Pickups, USPS Carrier Pickup v3).
  • Filing the drafted claim straight into the carrier's and the marketplace's dispute forms.

Built With

Share this project:

Updates

Submission history