YardSight AI

Autonomous Physical AI logistics vision OS: sub-100ms Dahua NVR P2P streaming, OpenCV 4.10 homography ANPR, hydrocarbon leak detection, and spatial VLM copilot powered by Nebius & NVIDIA Nemotron.

Comment

Inspiration

Commercial freight godowns, intermodal container terminals, and manufacturing distribution hubs operate under relentless throughput demands. Every day, thousands of 80,000-lb semi-trucks, dry vans, and chemical tankers enter and exit distribution centers. Yet beneath modern supply chains lies a multi-billion-dollar operational crisis:

  1. Gate Chokepoints & Demurrage Penalties: Truck drivers lose 6 to 14 hours waiting at facility entrance gates due to manual paper manifest entry, security guard phone calls, and manual inspection. At commercial detention rates of $75 to $85 per hour, shippers and carriers bleed over $1.2B annually in avoidable detention chargebacks while drivers' legal Hours of Service (HOS) expire.
  2. The Extreme Optical Angle Problem: Security cameras at gate barriers and perimeter towers are mounted high above ground (12 to 20 feet) for vandalism protection and wide coverage. As a result, incoming vehicles are viewed at steep 30° to 50° pitch and yaw angles. Standard commercial OCR and off-the-shelf vision systems fail completely on these distorted trapezoidal plates, forcing guards to step into traffic to manually transcribe numbers.
  3. Undetected Hazardous Chemical & Fuel Leaks: Heavy tractor-trailers frequently suffer blown hydraulic hoses or leaking diesel tanks on slick, polished warehouse concrete. Traditional cameras cannot distinguish between benign rainwater puddles and high-viscosity diesel spills. Unnoticed hydrocarbon spills cause severe forklift slide collisions, worker slip-and-fall injuries, and steep OSHA 1910.120 and EPA hazardous contamination fines.
  4. The Cloud AI Cost & Bandwidth Barrier: Modern logistics depots are outfitted with 16 to 64 legacy IP CCTV cameras (Dahua, Hikvision, Axis, ONVIF). Streaming 32 continuous RTSP 1080p feeds to hyperscaler cloud vision APIs exhausts facility uplink bandwidth (requiring >150 Mbps) and costs tens of thousands of dollars per month in cloud vision ingress fees.

YardSight AI was born to solve these physical challenges. We set out to build an autonomous, edge-native Physical AI operating system (GodownOS) that overlays directly onto legacy industrial CCTV infrastructure without requiring expensive sensor replacements, specialized hardware, or prohibitive cloud streaming bills.

What it does

YardSight AI transforms commodity industrial cameras into an intelligent, autonomous facility command system through five integrated capabilities:

1. 32-Channel Industrial CCTV Ingestion with Sub-100ms Latency

  • Proprietary Dahua PTCP Protocol Tunnel: Instead of relying on slow cloud REST scraping or RTSP port-forwarding, YardSight establishes a direct, full-duplex P2P UDP tunnel (dhp2p/tunnel.py) with Dahua signaling servers (www.easy4ipcloud.com:8800). It exposes an authenticated local loopback proxy on 127.0.0.1:1554.
  • Integrated WebRTC Gateway: Through embedded go2rtc integration, RTSP video is converted into sub-100ms WebRTC streams for zero-latency browser rendering.
  • Dual-Plane WebSocket Multiplexing: To bypass the browser's HTTP/1.1 6-connection pool limit, YardSight implements a high-frequency binary WebSocket multiplexer delivering 25–30 FPS video across 4, 8, 9, 16, or 32 camera tiles simultaneously with sub-20ms frame delivery in memory.
  • Strict Hardware Channel Binding: Every physical NVR channel (BE124E0PCA95532) is strictly mapped to its physical camera feed, preventing camera hopping or duplicate streams.

2. OpenCV 4.10 Perspective Homography ANPR Engine

  • Mathematical Projective Rectification: When a vehicle enters the gate at an acute 30°–50° angle, YardSight locates the four corner coordinates of the license plate boundary and computes the 3x3 projective transformation matrix using cv2.getPerspectiveTransform() and cv2.warpPerspective(), unwarping the distorted plate into a standardized orthogonal 400x120 planar pixel strip.
  • Image Enhancement Pipeline: The unwarped plate undergoes bilateral filtering, Contrast Limited Adaptive Histogram Equalization (CLAHE), and adaptive Gaussian thresholding, boosting OCR extraction accuracy above 96.4%.
  • Dual-Jurisdiction Logistics Support: Validates commercial USDOT freight carriers (TX 942-WKY, IL K88-2940) and Indian High Security Registration Plates (MH 12 RN 4589, DL 01 AB 9021).

3. Concrete Specular Reflectance & Hydrocarbon Leak Physics

  • Thin-Film Optical Dispersion Physics: Hazardous hydrocarbons exhibit a high refractive index ($n \approx 1.46 - 1.48$) and distinct iridescent chromatic dispersion. Rainwater has a lower refractive index ($n = 1.333$) and uniform diffuse reflectance.
  • Dual-Threshold Detection: Measures localized diffuse albedo drops (concrete darkening from fluid absorption) coupled with sharp specular glare peaks from overhead lighting.
  • Automated Hazardous Remediation: Calculates spill area in square meters ($m^2$), predicts leakage rate, and computes granular absorbent requirements under OSHA 1910.120 and EPA standards.
  • Interactive Rain / Moisture Slider: Allows facility operators to adapt sensitivity during rainstorms ($0.0 - 1.0$) to eliminate false positives.

4. Multimodal Spatial Reasoning with Nebius Token Factory & NVIDIA Nemotron

  • Cloud Intelligence Integration: Connects to the Nebius Token Factory API utilizing the open-source nvidia/Llama-3.1-Nemotron-70B-Instruct-HF model.
  • Multimodal Visual Prompting: Ingests live CCTV frame snapshots paired with real-time telemetry (gate queue, weighbridge weight, bay timers).
  • Natural Language Spatial Copilot (Nimbus): Operators can query dock demurrage risks, PPE safety compliance, and fluid accumulation in plain English.
  • Deterministic Offline Fallback: Seamlessly activates its edge spatial reasoning engine if cloud internet is interrupted.

5. Unified Logistics Operating System & Enterprise Workflows

  • 10 Core Workspaces: Logistics Command, Live CCTV Matrix (32-ch), Nimbus AI Hub, OpenCV 4.10 ANPR, Nebius VLM Agent, Concrete Specular Leaks, FactoryOS & Energy OCR, Fleet & Dock Dwell (8-bay digital twin), E-Way Bills & eBOL, and Scale Governance.
  • Multi-Channel Emergency Dispatch: Automated incident alerts dispatched via WhatsApp Cloud API and Twilio SMS.
  • 1-Click Cryptographic Audit Reports: Generates vector PDF compliance certificates via jsPDF for federal environmental and DOT audit filings.

How we built it

YardSight AI follows an edge-first, decoupled microservice architecture:

┌────────────────────────────────────────────────────────────────────────┐
│                        YARDSIGHT AI ARCHITECTURE                       │
├────────────────────────────────────────────────────────────────────────┤
│  [Physical Dahua NVR] ──(P2P UDP 8800)──► [dhp2p Signaling & Tunnel]   │
│         │                                        │                     │
│  [32 Optical Cameras]                    [RTSP Loopback 1554]          │
│         │                                        │                     │
│         ▼                                        ▼                     │
│  [OpenCV 4.10 Edge Vision]              [Embedded go2rtc Engine]       │
│   ├── Perspective Homography ANPR                │                     │
│   ├── Thin-Film Leak Physics                     ├── WebRTC (<100ms)   │
│   └── YOLOv8 / HOG Safety Tracking               └── Binary WS (25 FPS)│
│         │                                                │             │
│         ▼                                                ▼             │
│  [FastAPI Backend Server (8001)] ◄───────────── [React 19 Frontend]    │
│   ├── Demurrage Billing Engine ($85/hr)          ├── 32-Ch Matrix      │
│   ├── Dual US eBOL / India GST E-Way Bill        ├── Nimbus AI Hub     │
│   └── Multi-Tenant CASL RBAC / SQLite            └── 10 Workspaces     │
│         │                                                              │
│         ▼                                                              │
│  [Nebius Token Factory Studio]                                         │
│   └── nvidia/Llama-3.1-Nemotron-70B-Instruct-HF                        │
│       ├── Multimodal Visual QA (CCTV Frames)                           │
│       └── Spatial-Temporal Reasoning & Demurrage Mitigation            │
└────────────────────────────────────────────────────────────────────────┘
  • Cloud AI Backbone: Nebius Token Factory (https://api.studio.nebius.ai/v1)
  • Open-Source Model: nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
  • Edge Vision & Optics: OpenCV 4.10 (Python C-bindings), YOLOv8, HOG
  • Hardware Protocol: Dahua PTCP UDP P2P Reverse Engineering (dhp2p/tunnel.py)
  • Streaming Gateway: Embedded go2rtc, WebRTC, binary WebSocket multiplexer
  • Backend API: FastAPI (Python 3.10), SQLite, SQLAlchemy, Pydantic V2
  • Frontend OS: React 19, TypeScript, Vite, Tailwind CSS, Lucide Icons, jsPDF
  • Test Suite: 70/70 passing automated tests (56 backend pytest + 14 frontend vitest)

Challenges we ran into

  1. UDP Socket Saturation on Dahua Cloud Relay:

    • Challenge: Opening 32 simultaneous RTSP streams over a single P2P cloud relay socket saturated the UDP socket buffer, leading to dropped packets, video tearing, and NVR disconnections.
    • Solution: We engineered an active staggered channel worker (dhp2p/worker.py) that introduces a 500ms delay between channel initializations, prioritizes physical camera streams, and serves lightweight standby diagnostic cards for unpopulated slots.

  2. OpenCV VideoCapture Buffer Latency Accumulation:

    • Challenge: Standard cv2.VideoCapture.read() queues frames in FFmpeg's internal socket buffer. When processing frames sequentially, this caused real-world video display to lag behind by 3 to 10 seconds.
    • Solution: We implemented non-blocking socket buffer draining using cap.grab() in a continuous background thread, calling cap.retrieve() only when a consumer requests a fresh frame. This dropped latency from ~5,000ms down to sub-20ms in memory.

  3. Browser HTTP/1.1 6-Connection Pool Starvation:

    • Challenge: Modern web browsers enforce a strict limit of 6 concurrent persistent HTTP connections per domain. When rendering a 16-channel or 32-channel surveillance matrix, tiles beyond channel 6 hung indefinitely.
    • Solution: We built a dual-plane streaming architecture: primary feeds stream over peer-to-peer WebRTC via go2rtc, while matrix thumbnail grids stream binary JPEG frames over a single multiplexed WebSocket connection (/ws/cctv/stream), bypassing the HTTP connection limit entirely.

  4. Severe Camera Angle Homography Distortion:

    • Challenge: Security cameras mounted at high elevations view truck plates at acute 30°–50° angles. Simple affine transformations sheared characters and produced unreadable OCR.
    • Solution: We implemented a 4-point projective homography transformation (3x3 matrix) that maps the arbitrary trapezoid to a canonical 400x120 rectangle. We paired this with an interactive UI canvas where operators can fine-tune corner coordinates with real-time bilinear unwarping feedback.

  5. Differentiating Hydrocarbon Leaks from Rainwater:

    • Challenge: On polished industrial concrete, both water puddles and engine oil puddles appear as dark patches under standard grayscale thresholding, causing massive false-alarm spikes during rainstorms.
    • Solution: We integrated optical thin-film interference physics: measuring chromatic dispersion gradients and sharp specular reflectance peaks ($I > 230$) resulting from the high refractive index of hydrocarbons ($n \approx 1.47$) compared to water ($n = 1.33$). We also introduced an operator rain intensity slider ($0.0 - 1.0$) that dynamically raises the diffuse albedo threshold during downpours.

  6. Cloud Token Latency vs. Edge Gate Deadlines:

    • Challenge: Querying a 70B parameter vision-language model in the cloud takes 500ms to 2,000ms—unacceptable for an automated gate boom barrier that must raise in under 100ms.
    • Solution: We decoupled the architecture: high-speed classical OpenCV homography and OCR execute locally on the edge in sub-20ms to actuate gates, while the Nebius Token Factory and NVIDIA Nemotron handle asynchronous spatial-temporal reasoning, security audit synthesis, and safety compliance.

Accomplishments that we're proud of

  • 100% Passing Test Suite: Verified 70 out of 70 automated tests (56 backend pytest unit and integration tests + 14 frontend vitest tests) across all mathematical engines, security layers, and API endpoints.
  • Sub-20ms Frame Ingestion & Zero Frame Duplication: True 100% unique camera perspectives bound directly to physical Dahua NVR channels (BE124E0PCA95532), delivering fluid 25–30 FPS video under 5% CPU utilization and 250 MB RAM on commodity hardware.
  • Deep Physical Hardware Integration: Successfully built a native P2P PTCP network tunnel directly to industrial Dahua security hardware without relying on proprietary cloud web clients.
  • Seamless Dual-Market Logistics Execution: Complete, production-grade support for both North American interstate trucking (USDOT carriers, state plates, eBOL) and Indian industrial corridors (HSRP plates, GST E-Way Bill Form EWB-01, WhatsApp dispatch).
  • Physics-Grounded Environmental Protection: Solved a critical real-world industrial safety hazard by applying thin-film optical physics to detect toxic hydrocarbon leaks and calculate OSHA absorbent boom requirements.
  • Production-Grade Demo Video with Voiceover: Generated a complete 1080p, 2m 57s walkthrough video featuring clear neural audio narration (en-US-ChristopherNeural), demonstrating genuine physical camera feeds and UI workflows.

What we learned

  1. Classical Computer Vision and VLMs are Complementary, Not Competitive: While large vision models like Nemotron 70B excel at semantic scene reasoning, common-sense deduction, and spatial relationships, classical algorithms (perspective homography, bilateral filters, CLAHE, Otsu thresholding) remain unmatched for deterministic, microsecond-level geometric tasks. The most powerful Physical AI systems combine both.
  2. Industrial Edge Networking Demands Resilience: Legacy CCTV cameras on factory floors and warehouse perimeters operate in hostile networking environments: high packet loss, cellular backhauls, and strict NAT firewalls. Building packet reordering buffers and deterministic offline fallback cards is essential for 24/7 industrial uptime.
  3. The Power of Open Models on Nebius: Using nvidia/Llama-3.1-Nemotron-70B-Instruct-HF hosted on Nebius Token Factory delivered exceptional reasoning quality with conversational latencies that made live facility copilot interactions feel natural and responsive.

What's next for YardSight AI

  • Direct Modbus & MQTT Gate Barrier Actuation: Integrating directly with physical PLC controllers and industrial relay boards (Advantech, Moxa) to automatically trigger gate boom barriers upon verified ANPR manifest match.
  • Edge Cluster Mesh Networking: Deploying YardSight AI instances on NVIDIA Jetson Orin Nano micro-servers at each gate, federating telemetry across multiple regional godowns into a single enterprise dashboard.
  • Autonomous Drone Patrol Integration: Ingesting RTSP video feeds from autonomous yard security drones (Skydio, DJI Enterprise) to monitor rooftop integrity and inspect exterior trailer seals.
  • NVIDIA Cosmos World Model Exploration: Exploring NVIDIA Cosmos physics-informed world models to simulate complex truck maneuverability in tight container godowns and predict yard traffic gridlocks before they form.

Built With

Share this project:

Updates

Submission history