Inspiration

A dog walks into a shelter. Someone glances at it for a few seconds and writes "pit mix." That guess goes into the record and follows the dog for its entire stay, shaping who stops at its kennel and how long it waits. Yet research on dog genetics shows breed explains only a small share of how an individual dog actually behaves.

Shelters already have cameras on their kennels. We asked: what if we judged shelter dogs by what they do, not what we think they are?

What it does

Bone-a-find turns kennel video into a breed-free behavior record for every dog.

  • Live video analysis: staff upload a kennel clip; the pipeline finds the dog, tracks it, and scores energy, stress and approachability.
  • Every score has receipts: each behavior event on the evidence timeline links to the exact second of video. Click "repetitive pacing" and the video jumps there. Staff confirm, reject, or add context.
  • Behavior over time: readings are stored as a time series, so we catch dogs who are struggling. Lola's rising stress triggers a welfare alert; after staff moved her to a quieter wing, her readings came down. (We show the change; we don't claim causation.)
  • Whole-shelter matching: every dog is matched against every family at once with the Hungarian algorithm, with plain-English reasons for each match. Dogs who need extra support are routed to the behavior team instead of placed.
  • Family intake: a Gemini-powered counselor that remembers returning families.
  • Ask about a dog: answers like "Is Lola good with kids?" drawn from volunteer notes, citing the day.
  • Public Behavior Passport: a printable, scannable, breed-free report written by AWS Bedrock.
  • Housing help: a TinyFish agent finds real pet-friendly rentals that will accept the dog, because housing is a common reason adoptions fall through.
  • Bias check: the same dog under different breed labels. A general-purpose model's description shifts with the label; our scores stay identical because breed never enters our pipeline.

How we built it

  • Frontend: Next.js on AWS Amplify, served at bone-a-find.us.
  • Video: browser uploads go straight to Amazon S3 via presigned URLs (Amplify caps request bodies at about 6 MB, so video can't pass through the app server).
  • Behavior pipeline: FastAPI on Amazon EC2 behind Caddy (HTTPS). YOLO11 for detection, ByteTrack for multi-frame tracking, camera-shake compensation, then behavior extraction. Videos are transcoded to H.264 so they play in any browser.
  • Data: MongoDB Atlas for dogs, families, jobs and matches; Tiger Data for longitudinal welfare readings.
  • AI services: AWS Bedrock (Nova Lite) for grounded passports and summaries, Gemini for family intake and the bias experiment, Supermemory for grounded staff knowledge, TinyFish agents for live web tasks.
  • Matching: we build an all-dogs × all-families compatibility matrix and solve the assignment problem:

$$\max_{\sigma} \sum_{i} C_{i,\sigma(i)}$$

where \( C_{ij} \) is the compatibility of dog \( i \) with family \( j \), using the Hungarian algorithm, so the shelter gets the best overall placements rather than greedy first-come matches.

Responsible AI

  • Breed is never recorded or used. It's not hidden or toggled off; it simply isn't an input.
  • Humans decide. Every score is decision support, not a diagnosis. Staff verify, and corrections are kept.
  • No medical diagnoses, permanent temperament claims, or safety certifications.
  • Confidence-gated experiments. We prototyped DeepLabCut SuperAnimal-Quadruped pose estimation (39 keypoints) on 9 shelter videos. Model confidence reached about 0.65–0.78 on 6 of 9 clips, but it isn't validated yet, so the system refuses to let pose influence matches until it passes our gate.

Honest numbers

  • 86.7% agreement (13/15) with our written expected-behavior bands on a small evaluation set. This is agreement, not "accuracy."
  • Deterministic: maximum score difference of 1 across repeated runs.
  • Bias check: the general-purpose model's output varied with the supplied breed label, within its own run-to-run spread (up to about 6.7 points); our scores were identical across every label.

Challenges we ran into

  • Moving video through the cloud: Amplify's request limit forced a presigned direct-to-S3 upload, and the S3 bucket and keys had to live in the right AWS account for both the web app and the EC2 pipeline.
  • Environment variables at build time: Next.js bakes some values in during the build, so new settings didn't take effect until we triggered a fresh build.
  • Production infrastructure overnight: systemd for a persistent API, Caddy for automatic HTTPS, DNS for a custom domain, and wiring Mongo and Tiger into the server.
  • Real kennel footage is hard: bars, odd angles and lighting cut pose-model confidence, which is exactly why we built a gate instead of trusting it.
  • Browser playback: mp4v video wouldn't play in browsers, so we transcode to H.264.

What we learned

  • Removing bias after the fact is harder than designing it out from the start.
  • Evidence beats scores: people trust a number when they can click it and see why.
  • An AI feature that knows when to stay quiet is a feature, not a gap.
  • Shipping to real cloud infrastructure teaches more than any localhost demo.

What's next

  • Calibrate the pose layer against volunteer-labeled posture, then enable it behind the gate.
  • Pilot with a local shelter and measure length of stay, return rates and staff hours saved.
  • Integrate with existing shelter management software.
  • Business model: free for small rescues, per-shelter SaaS for larger shelters and county contracts.

Every dog deserves to be judged by what it does, not what it looks like.

Built With

Share this project:

Updates

Submission history