PawTrace

Inspiration

When a pet goes missing, every minute matters, but useful information is often scattered across neighbourhood posts, doorbell cameras, shelter reports, and word of mouth. Owners are left manually sorting through incomplete sightings without knowing where to search next. We built PawTrace to bring those signals together and turn them into a focused, evidence-based search.

What it does

PawTrace combines community sightings with footage from doorbell cameras, home cameras, and local businesses to determine where a missing pet is most likely to be.

It compares descriptions and images while considering when and where each sighting occurred. PawTrace then creates a prioritized search map, identifies nearby cameras that may have captured the pet, analyzes their footage, and follows the pet’s estimated direction of travel. When a possible match is found, the owner sees the camera frame alongside their reference photos and makes the final decision.

How we built it

We built the frontend with Next.js, TypeScript, Tailwind CSS, and React Leaflet, with a FastAPI backend connected to Elasticsearch.

Elasticsearch acts as PawTrace’s decision engine. We combine BM25 keyword search, text-vector search, image-vector search, and geographic and time-based scoring through reciprocal rank fusion. Credible sightings are used to update a hexagonal search map based on distance, terrain, movement, and previously searched areas.

Camera fields of view are stored as geographic shapes, allowing us to query which cameras overlap the highest-priority areas. Camera footage is processed using YOLOv8 for detection and tracking, while DINOv2 compares detected animals with the owner’s reference photos. An agent with 13 tools coordinates the search, camera selection, and evidence gathering, with its activity streamed to the interface using Server-Sent Events.

Challenges we ran into

One of our biggest challenges was tuning Elasticsearch to return confident matches without filtering out useful sightings. A visually similar dog might have the wrong collar, while a nearby report might be too old to remain relevant. We repeatedly adjusted the retrieval weights, geographic and time-decay parameters, RRF settings, confidence threshold, and contradiction penalties until the system balanced recall with accuracy. We tested it against similar breeds, missing information, outdated reports, and impossible travel distances to make sure a single strong signal could not overpower conflicting evidence.

Accomplishments that we're proud of

We are proud that PawTrace produces explainable results instead of blindly returning the closest visual match. Every result shows how its appearance, description, location, time, and source reliability affected its final score. After tuning the Elasticsearch pipeline, PawTrace correctly rejects unrelated dogs, scores a similar-looking dog only 0.30 when its collar contradicts the report, and returns no confident match for an unseen pet. In our main demonstration, it combines several credible signals to reduce the immediate search area from 1.29 square kilometres to 0.10 square kilometres.

What we learned

We learned how lexical, semantic, visual, geographic, and time-based search can complement one another when no individual signal is reliable enough by itself. We also learned how to model camera visibility with geographic shapes, estimate real-world movement from video, and apply search-theory concepts to prioritize an area.

Most importantly, we learned that an agent becomes much more useful when it has carefully limited tools, explainable decisions, and clear boundaries around actions that require human confirmation.

What's next for PawTrace

The next step is to pilot PawTrace with local pet-rescue groups, shelters, and willing camera owners. We would add opt-in camera integrations, live shelter data, SMS alerts, and privacy-preserving video processing that keeps footage on the camera owner’s device whenever possible.

We also want to improve movement prediction using additional real-world cases, support more animal types, and allow volunteers to mark searched areas from their phones so the search map updates in real time.

Built With

• elasticsearch
• fastapi
• nextjs
• node.js
• python