Aegis Vision AI

An autonomous Physical AI and Agentic Vision platform powered by OpenCV 5 and AWS that transforms passive industrial monitoring into an autonomous See-Reason-Act loop.

Comment

Inspiration

Every year, industrial manufacturing and semiconductor facilities suffer tens of billions of dollars in losses from machine accidents, hazardous chemical leaks, and batch contamination. Most computer vision systems in factories are passive observers: they record video or flag an alert on a dashboard, leaving mitigation to distracted human operators who often react seconds or minutes too late.

That problem led to a question: What if computer vision could close the physical feedback loop? What if OpenCV 5 didn't just see a hazard, but had an autonomous agentic brain that evaluated safety regulations (OSHA, ISO, NFPA) and directly actuated physical factory equipment—stopping conveyors, rerouting mobile robot fleets, sealing HVAC dampers, and syncing with AWS IoT Device Shadows in real time?

That question led to Aegis Vision AI.

What It Does

Aegis Vision AI is an end-to-end autonomous Physical AI platform operating across 4 critical industrial channels:

  1. CAM-01: Semiconductor Wafer Cleanroom Micro-Inspection:
    • Sub-pixel Canny edge detection, morphology, and circularity analysis detect micro-cracks and void ratios, computing real-time wafer die yields.
  2. CAM-02: Robotic Workcell & Dynamic Safety Zone:
    • Dense Farneback Optical Flow calculates arm motion vectors. Point-in-polygon intrusion checks detect workers entering active hazard zones and halts the machinery within 140ms.
  3. CAM-03: Hazardous Chemical Bay Spill Containment:
    • Multi-channel HSV/YCrCb fluid segmentation isolates toxic spills, calculates temporal expansion rate ($dA/dt$), engages negative-pressure HVAC scrubbers, and reroutes AGV robot fleets away from the hazard.
  4. CAM-04: Cleanroom Airlock PPE Compliance:
    • Verifies safety hardhat and high-visibility vest compliance against OSHA 1910.132 standards at security turnstiles.
  5. Autonomous Agentic Cognitive Engine:
    • Employs a See -> Reason -> Act loop that evaluates structured visual telemetry against authoritative safety policies, formulates multi-step mitigation plans, and dispatches physical actuators and AWS cloud events.
  6. COOL (Cloud-Optimized OpenCV Library) Studio:
    • Implements ARM64 NEON vectorization and cache-tiled streaming optimized for AWS Graviton3/4 processors, achieving a 3.5x–4.1x speedup and over 50% cloud cost reduction.
  7. Facility Digital Twin 2D Canvas & Compliance Exporter:
    • Live interactive map displaying active incident perimeters, camera cones, and animated AGVs, plus one-click cryptographic compliance audit report generation.

How I Built It

  • Computer Vision Core: Native OpenCV 5.0 in Python with headless streaming pipelines utilizing sub-pixel edge detection, Farneback optical flow, morphological operators, CLAHE, and multi-space color segmentation.
  • COOL Acceleration: Tiled memory-mapped buffers and SIMD / ARM NEON acceleration paths designed for AWS Graviton (ARM64 Neoverse) instances.
  • Agentic AI & Reasoning: Autonomous agent core integrating AWS Bedrock (Anthropic Claude 3.5 Sonnet) with local deterministic regulatory expert rules for OSHA 1910.212, ISO 14644-1, and NFPA 704.
  • Cloud Infrastructure: Amazon Web Services (AWS) via boto3 for S3 (forensic artifact storage), IoT Core (Device Shadow digital twin state), CloudWatch (latency metrics), and DynamoDB (incident ledger). Includes an auto-detecting high-fidelity local simulator for zero-config evaluation.
  • Frontend Mission Control: Built with modern Vanilla CSS, HTML5 Canvas, responsive dark-mode industrial design, glassmorphism, dynamic MJPEG live video streaming, and real-time WebSocket/REST polling.

Challenges I Ran Into

  • Latency in Closed-Loop Actuation: Real-world safety requires sub-150ms actuation from photon capture to machine stop. Standard OpenCV pipelines struggled under 4K frame loads. We solved this by developing the COOL engine, partitioning frames into vertical cache-coherent tiles matching AWS Graviton L2 cache limits, bringing frame latency down to ~1.0ms.
  • Agent Hallucination Mitigation: In safety-critical industrial settings, an LLM cannot be allowed to invent safety protocols. We anchored the Agentic Vision Engine in formal SOP rulebooks with strict schema validation.

Accomplishments I’m Proud Of

  • Achieving a verified 3.5x–4.1x performance speedup (71.8% latency reduction) using COOL principles on ARM64 Graviton architecture.
  • Building a true closed-loop Physical AI system where visual perception drives automated physical actuation.
  • Creating a zero-config, out-of-the-box experience where judges can test live video streams, webcam feeds, custom uploads, benchmarks, and digital twin animations without needing paid cloud setup.

What I Learned

  • How cache-line alignment and vector register sizing (ARM NEON 128-bit) directly impact cloud streaming economics.
  • How to bridge the gap between continuous pixel tensors (OpenCV) and discrete symbolic reasoning (Agentic LLMs).

What's Next

Next, I want to:

  • Integrating 3D Gaussian Splatting and spatial depth cameras (Intel RealSense / Stereolabs ZED 2i) into the OpenCV 5 pipeline.
  • Deploying AegisVision AI to edge micro-controllers (AWS IoT Greengrass on NVIDIA Jetson / Raspberry Pi 5).
  • Expanding the SOP knowledge base to include EU Machinery Directive 2006/42/EC and semiconductor SEMI S2 standards.

Built With

Share this project:

Updates

Submission history