ProofSight

## Inspiration

In field service, equipment rental, and industrial maintenance operations, kits of specialized tools must be verified before dispatch and upon return. 

Traditional visual inspection applications rely on a **one-shot approach**: a technician snaps a single overhead photograph, and a computer vision model attempts to classify every item. However, in physical environments, tools are frequently partially occluded by a glove, coiled test lead, or protective wrapping. Conventional models either hallucinate presence or trigger false missing-item alarms—halting operations and costing enterprises millions in unnecessary restocking cycles and downtime.

We realized that **computer vision shouldn't be a passive one-shot exam; it should be an active conversation between an autonomous agent and the physical world**. If the AI is uncertain about what it sees, it shouldn't guess—it should guide the operator to capture the missing visual evidence.

---

## What It Does

**ProofSight** is an **Autonomous Visual Evidence Agent** that turns any smartphone into an intelligent inspection station for equipment kits and reusable tool sets:

1. **Substantive OpenCV 5 Ingest Gate**: Intercepts unusable imagery before analysis. It computes the Laplacian blur variance:
   $$\sigma^2_{Lap} = \text{Var}(\nabla^2 I)$$
   If $\sigma^2_{Lap} < 100$, the image is rejected ($\text{GATE: BLUR REJECTED}$), preventing spurious classifications. It also evaluates exposure histograms to detect severe under- or overexposure.
2. **Separates "Unknown" from "Missing" (BR-03)**: When a tool is partially occluded, ProofSight marks the component as `UNRESOLVED` rather than triggering a false "missing item" rejection.
3. **Active 3D Inclinometer Guidance (BR-05)**: The agent calculates the expected uncertainty reduction:
   $$\mathbb{E}[\Delta U] > C_{\text{capture}}$$
   When positive, the HUD projects an interactive 3D Inclinometer and Artificial Horizon (`TARGET 45° // ▲ TILT +25°`), directing the technician to the exact glancing angle needed to view beneath the occlusion.
4. **Incremental Multi-Frame Evidence Accumulation (BR-13)**: Secondary captures verify the targeted component while mathematically preserving previously verified items without restarting the inspection.
5. **Deterministic Policy Release (BR-09)**: An isolated Hexagonal policy engine enforces 15 strict business invariants. Only when all mandatory components are verified does the system authorize release, stamping the session with an immutable SHA-256 evidence certificate digest.
6. **Supervisor Exception Terminal (BR-10)**: If a tool is genuinely absent, an authorized supervisor can sign off via a tactical override terminal, recording reviewer identity and justification permanently in the audit log.

---

## How We Built It

We engineered ProofSight using a strict **Hexagonal Architecture (Ports & Adapters)** and **Domain-Driven Design (DDD)**:

- **Pure Domain Core (`src/domain/`)**: Built in pure JavaScript with zero external runtime dependencies. It manages an explicit Finite State Machine (`NOT_OBSERVED` $\rightarrow$ `UNRESOLVED` $\rightarrow$ `VERIFIED` / `SUSPECTED_MISSING`), audit trails, and the deterministic policy engine.
- **Computer Vision Pipeline (`server/`)**: Powered by **OpenCV 5** and **FastAPI**. It handles Laplacian blur variance, luminance histograms, Canny edge density per region-of-interest, and ORB feature matching with homography perspective correction:
  $$p' \sim H \cdot p$$
- **Frontline Inspection Cockpit (`src/ui/`)**: A responsive mobile-first React 18 & Vite web application featuring:
  - Interactive spatial AR target acquisition reticles with chamfered geometry.
  - 3D Inclinometer & Artificial Horizon HUD for angle guidance.
  - Digital Twin Manifest with custom SVG tool schematics and live confidence gauges.
  - Knurled titanium shutter with radial SVG progress rings and aperture animations.
  - Real-time acoustic feedback synthesized via the native **Web Audio API**.
- **Containerization & Cloud**: Packaged via multi-stage Dockerfiles and `docker-compose.yml`, ready for **AWS App Runner** or **AWS ECS Fargate**.
- **Automated Verification**: 21 native unit tests, 3 Playwright slow-motion E2E acceptance test suites, and an Affirmation Matrix guaranteeing 100% bidirectional traceability across 173 atomic requirements.

---

## Challenges We Faced

1. **Escaping the "One-Shot Fallacy"**: Transitioning from traditional single-image classification to an incremental, multi-frame state machine required careful domain isolation so that secondary observations never overwrote valid existing proofs.
2. **Ergonomics for Field Technicians**: Frontline workers cannot navigate complex web forms or read raw CV confidence numbers while wearing gloves. We iterated heavily to create a high-contrast, tactile cockpit layout that consolidates the camera viewfinder, spatial AR reticles, and shutter trigger into a single thumb-accessible station.
3. **Deterministic Safety Guarantees**: Probabilistic computer vision outputs can never be allowed to directly approve safety-critical releases. We solved this by enforcing a hard separation between perception (OpenCV measurements) and decision authorization (deterministic business rule engine).

---

## Accomplishments That We're Proud Of

- **100% Occlusion Resolution**: In our 30-scenario benchmark, ProofSight resolved 12 of 12 partially occluded tools through active angle guidance, cutting false missing-item rejections from **37.5% down to 0.0%**.
- **Zero Hallucinated Approvals**: Invariants mathematically bar approval if any mandatory item lacks verifiable evidence.
- **Sub-30ms OpenCV Execution**: The full OpenCV 5 quality, exposure, and feature pipeline runs in under 25ms on AWS Graviton3.
- **100% Verified Traceability**: 173 of 173 atomic requirements pass our automated 95% Plan Affirmation Gate with 0 orphan assertions.

---

## What We Learned

We learned that in frontline industrial operations, the highest-leverage application of AI is not generating broad descriptive reports, but **actively guiding the operator to capture better physical evidence**. Providing clear, directional spatial instructions transforms computer vision from a brittle guessing tool into a reliable operational partner.

---

## What's Next for ProofSight

- **Hardware IMU Integration**: Binding the 3D Inclinometer directly to the mobile device's accelerometer (`DeviceOrientationEvent`) for physical gyroscope lock-on in 3D space.
- **Custom Template Studio**: A visual tool allowing operations managers to photograph new equipment trays and automatically generate reference coordinate templates.
- **Enterprise Integrations**: Native connectors for SAP Plant Maintenance and IBM Maximo to trigger automated work orders and digital asset handovers.

Built With

• agentic-vision
• amazon-web-services
• artificial-intelligence
• bash
• computer-vision
• css3
• docker
• domain-driven-design
• fastapi
• git
• hexagonal-architecture
• html5
• javascript
• node.js
• numpy
• opencv
• playwright
• pydantic
• python
• react
• rest-api
• uvicorn
• vite
• web-audio-api