Sentry

Try this image in the fly dev website if you want to see it work live!

Inspiration

Quality escapes quietly eat small manufacturers alive. ASQ pegs quality-related costs at 15 to 20 percent of sales revenue, and the classic 1-10-100 rule says every stage a defect survives multiplies its cost by ten: a dollar at prevention, ten on the line, a hundred once it ships. The tooling that solves this exists, but inline AOI machines run $30k to $200k+ with another $20k to $40k in first-year installation, training and maintenance, while Quality Magazine's spending survey found 30 percent of plants budget under $10k for optical inspection. So small shops inspect by eye, and the research on that is brutal: Sandia's human-factors study found even elite trained inspectors catch only 85 percent of defects while false-rejecting 35 percent of good parts, close to the commonly cited 80 percent industry average. We built Sentry to close that gap: factory-grade visual anomaly detection from a single photo, at a price of roughly zero.

What it does

Upload a photo of a manufactured part (our demo line is PCBs). Sentry registers it to a known-good golden reference with ORB feature matching and RANSAC homography, scores it with a PatchCore anomaly model trained ONLY on photos of healthy units, localizes anything off-distribution, and draws a tight red contour around the actual fault on your original photo. A vision-language model then compares the flagged crop against the same region of the reference board and describes the physical problem in one technician sentence: "the battery is missing," "corner of the board is broken off." On a defect verdict, Sentry phones the line operator: a synthesized voice reads the case, then hands off to an interactive agent that answers questions. On the page, an embedded copilot chat answers anything about the current board, per-component defect history, and shift stats, grounded entirely in tool calls over the real inspection record. No defect examples are ever needed for training, which matters because real production lines have thousands of photos of good parts and almost none of failures.

How we built it

The inspection core is Python: FastAPI serving a PatchCore model (anomalib) that builds a coreset memory bank of wide_resnet50 features from ~80 photos of healthy boards, trained in minutes on CPU. OpenCV handles golden-board registration so phone photos at any angle map onto the reference frame. Every inspection streams over a websocket hub, lands in a case record with sqlite history, and renders as contours on the original image.

ElevenLabs powers the operator call voice. When a board fails, we synthesize the case summary with ElevenLabs TTS (turbo v2.5), cache the MP3 by content hash, and serve it to Twilio's call flow, so the operator hears a natural voice reading board ID, findings and disposition instead of a robo-voice. The fallback chain is real engineering: if synthesis is unavailable the call still goes out with standard TTS.

CopilotKit is the entire in-app copilot surface. A CopilotKit runtime gateways to the LLM, and the chat panel on the station page defines the inspector tools as frontend actions: latest case, per-designator defect history, shift summary, escalate, quarantine batch. The live on-screen inspection result streams into the copilot's context with useCopilotReadable, so "why did this board fail" is answered from what's actually on screen, and every claim the copilot makes traces to a tool result.

Fireworks AI trains our defect description model on our specific boards. We wrote a Python script that takes our ~80 PCB photos and defect crops, pairs each one with a one-sentence technician description, and packages it all into Fireworks' JSONL dataset format. Upload the dataset, kick off a LoRA fine-tune of Qwen2.5 VL, and about 20 to 30 minutes later the tuned model deploys with one click to the same endpoint our pipeline already calls. Small models tune free on Fireworks, so the entire loop cost nothing.

Daytona gave us clean, disposable sandboxes for tuning the model. Every experiment (coreset ratio, threshold, gate strictness) spun up a fresh sandbox through the Python SDK, trained the memory bank, scored the eval set, and sent back the results. We reviewed the flagged overlays by hand, marked false positives, and fed that human feedback into the next run. Because every run started from an identical clean environment, when a config won we knew it was the config and not stale state on somebody's laptop.

Braintrust kept us honest about whether any of that tuning actually helped. Every inspection run logs as an eval: verdicts score against ground truth we labeled by hand, and any two experiments diff side by side in their UI. So when we changed a threshold or tightened the vision gate, we saw exact precision and false-positive numbers against the previous baseline instead of eyeballing a few overlays and calling it better. Daytona ran the experiments, Braintrust kept score, and together they turned tuning into a loop we could trust.

Challenges we ran into

PatchCore has zero rotation invariance, and real uploads arrive at every angle: the fix was full golden-board registration, ORB plus RANSAC homography, with an orientation-search fallback, then mapping every overlay back through the inverse transform so contours land on YOUR photo, not the normalized one. Stickers, tape and lighting kept triggering false positives, so we added a comparative vision gate: the model sees the flagged crop next to the same region of a known-good board and must distinguish damage from cosmetic difference, with hedged answers automatically downgraded. Twilio trial accounts silently reject inline call instructions, so TwiML has to be served from a public tunnel. And CopilotKit's server-side actions in the current release execute the tool but end the run before the model can speak, which we solved by moving all tools client-side as frontend actions, where the round-trip machinery is battle-tested.

Accomplishments that we're proud of

The detector matches human ground truth: on our demo boards, the contour Sentry draws around a missing battery matches the circle a human drew by hand, and a broken corner and burnt region are each caught and described correctly in one sentence. The whole verdict-to-phone-call loop runs unattended. And the memory bank trains from nothing but healthy-board photos in minutes on a laptop CPU, which is the entire economic point.

What we learned

Anomaly detection beats classification for real factories: you can ship defect detection having never seen a defect. Registration is most of the battle for camera-in-the-wild inspection; the model is only as good as the alignment in front of it. And voice and chat agents become trustworthy the moment every answer is forced through tools over real records instead of generation.

What's next for Sentry

PCBs were our demo line, but nothing in Sentry is PCB-specific, and that is the whole plan. Because the model trains on photos of healthy units alone, onboarding a new part is just pointing a camera at good product for a shift: machined parts, injection-molded housings, stamped metal, bottled goods, textiles, food packaging. We want that onboarding to be fully automatic, where a shop uploads or streams photos of any object, Sentry builds the memory bank on its own, and inspection is live the same day with zero ML expertise required.

The next step is hardware: a small plug-and-play device that connects to a simple off-the-shelf camera mounted over the line, runs the inspection loop at the edge, and phones out on defects. No $100k AOI machine, no integrator, just a box and a camera clamped above the conveyor. From there, multi-workspace support lets one company run separate stations, lines and facilities under one roof, each with its own golden references and defect history but shared dashboards and fleet-wide learning, so a defect pattern caught at one station makes every station smarter.

Industry-wise, we are starting with the shops the AOI market prices out: low-volume, high-mix electronics, then adjacent manufacturing verticals like plastics, metal fabrication, automotive subcontractors, medical device assembly, and food and beverage packaging, anywhere a human is currently squinting at parts all day. Alongside that, growing the fine-tuning dataset with every labeled defect crop the fleet produces so the description model keeps improving from live production data, and a ticketing sink for escalations.