Inspiration
60% of new restaurants fail their first health inspection. This results in an average 3-week opening delay, $8,400 in lost revenue, and thousands in emergency contractor fees. Health codes are 200+ page legal documents that vary by zip code, and small business owners—from food trucks to ghost kitchens—are flying blind against regulators.
We realized this isn't just a "knowledge" problem; it's a data retrieval and interpretation gap. We built Inspector Joe to level the playing field. We engineered an autonomous auditing system that gives the average business owner the power of a veteran health inspector, allowing them to "pre-fail" an inspection privately before it costs them their livelihood.
What it does
Inspector Joe is a multimodal compliance engine that performs real-time visual auditing. It creates a "digital twin" of a professional health inspector using a bespoke neuro-symbolic architecture.
- Spatial Violation Detection: Uses a custom-trained object detection model to perform semantic segmentation on the live video feed. It identifies non-compliant objects (e.g., cross-contamination vectors, improper storage) and renders precise bounding boxes around the exact violation.
- Regulatory RAG (Retrieval-Augmented Generation): The system uses a vector database to cross-reference visual findings with specific local health codes (e.g., "Texas Food Establishment Rules §228.70") rather than generic advice.
- Dynamic Inspection Pipelines: Instead of a static checklist, the system constructs a Directed Acyclic Graph (DAG) of inspection steps based on the specific business ontology.
- Financial Risk Modeling: It projects financial risk by estimating labor and material costs for compliance fixes based on current market rates.
How we built it
We architected Inspector Joe as a cascading inference pipeline utilizing a microservices architecture.
1. The Perception Layer (Custom Vision Pipeline) We moved beyond off-the-shelf APIs and trained a custom YOLOv8 model specifically on health code violations (dirty sinks, uncovered food, improper chemical storage).
- Training Data: We curated a dataset of 5,000+ kitchen compliance images to fine-tune the model for high-precision detection in low-light environments.
- Inference Engine: The model runs via ONNX Runtime for optimized edge inference, ensuring sub-100ms latency.
- Coordinate Normalization: A custom geometry engine maps the model's bounding box outputs to the device's variable viewport aspect ratio in real-time.
2. The Cognitive Layer (RAG & Vector Search) We built a Retrieval-Augmented Generation (RAG) pipeline to handle the 50 distinct state health codes.
- Vector Database: We chunked and embedded legal texts from FDA and local health departments into Pinecone.
- Semantic Search: When a violation is detected (e.g., "sponge in sink"), the system queries the vector store for the relevant statute, retrieving the exact legal text (
Section 4-204.112) to display to the user. - LLM Synthesis: We use a large language model as a reasoning layer to synthesize the visual detection data with the retrieved legal text, ensuring the final report is legally accurate and context-aware.
3. Infrastructure & Edge Optimization
- Frontend: React 19 with a custom WebGPU-accelerated canvas layer for 60fps bounding box rendering.
- Backend: Python FastAPI orchestrating the neural search and inference requests.
- Database: Supabase (PostgreSQL) for relational data and user session persistence.
Challenges we ran into
The "Visual Grounding" Alignment Problem: Standard models are excellent at identifying generic objects ("sink") but struggle with context ("sink used for wrong purpose"). Training the YOLOv8 custom head to distinguish between a "prep sink" and a "handwash sink" based on surrounding objects was a significant data engineering challenge. We solved this by implementing a spatial relationship heuristic, analyzing the proximity of soap dispensers to classify the sink type before checking for violations.
Latency vs. Accuracy Trade-off: Chaining a vision model, a vector search, and an LLM created a latency bottleneck. We refactored the pipeline to be asynchronous: The vision analysis happens immediately on the edge, while the heavy RAG citation lookup streams in via a background thread (WebSocket), creating a perceived instant experience for the user.
Accomplishments that we're proud of
- Bridging Computer Vision and Legal Tech: We successfully integrated a custom object detection pipeline with a legal RAG system, allowing the AI to "see" a violation and "cite" the law simultaneously.
- Zero-Shot Generalization: Our custom model successfully identifies complex violations in novel kitchen layouts it wasn't explicitly trained on.
- Enterprise-Grade Architecture: The separation of concerns between the Vision Microservice (Execution) and the RAG Agent (Reasoning) makes the codebase modular and scalable.
What we learned
- Data Quality is King: Fine-tuning a vision model requires impeccably labeled data. We learned that "garbage in, garbage out" applies doubly when dealing with safety-critical compliance detection.
- Neuro-Symbolic AI: The future isn't just Deep Learning. It's combining Deep Learning (for vision) with Symbolic Logic (Vector RAG). Inspector Joe works because it respects hard rules (health codes) while using soft computing (vision) to perceive the world.
What's next for Inspector Joe
- Edge Deployment: Distilling the vision model to run via TensorFlow.js directly in the browser, removing the server round-trip entirely for privacy and speed.
- IoT Sensor Fusion: Integrating with Bluetooth LE temperature probes to combine visual data with thermal telemetry.
- Expansion: Scaling the RAG knowledge base to include OSHA and Fire Safety codes.
Built With
- docker
- fastapi
- openai-embeddings
- pinecone
- postgresql
- python
- pytorch
- react
- supabase
- typescript
- yolov8
