Orb

Autonomous OpenCV 5.0 agentic vision on AWS Graviton2 & S3 that perceives degraded scenes, auto-remediates low-light pixels via LAB CLAHE, and grounds multimodal AI in deterministic HUD telemetry.

Comment

Inspiration

Multimodal Vision-Language Models (VLMs) are remarkable at high-level storytelling, yet they frequently fail when confronted with real-world visual degradation. Feed a foundation model an underexposed inspection frame, a shadowed architectural capture, or a motion-blurred video clip, and it will confidently hallucinate spatial geometry, miss fine structural defects, and guess at focal boundaries without offering any pixel-level proof. Conversely, classical computer vision libraries produce exact geometric and photometric primitives edges, structural vanishing lines, frequency sharpness, and optical flow vectors but stop short of semantic reasoning or actionable synthesis.

We built Orb to bridge that exact gap: uniting the deterministic pixel-level precision of OpenCV 5.0 with an autonomous Perception → Decision → Action agentic loop deployed natively on AWS Graviton2 (arm64) and Amazon S3. Instead of blindly sending raw uploads to a black-box AI, Orb inspects the visual evidence first, autonomously repairs degraded low-light frames in CIE LAB space using OpenCV 5.0 before inference, and lets users visually verify every geometric detection through an interactive OpenCV Vision HUD.

What it does

Orb is an autonomous visual intelligence and multimodal scene-analysis platform built for visual inspection engineers, cinematographers, digital creators, and decentralized compute users.

Live Judge Evaluation Endpoint (10 Pre-Allocated Credits, Zero Wallet Setup): https://orb-azure-ten.vercel.app/?judge=OPENCV2026 Direct AWS Graviton2 (arm64) Lambda Health Endpoints: https://jsnwiqkw5sqs3c7hag6ywd3ngm0vljpc.lambda-url.us-east-1.on.aws/health (GET /health & GET /api/health) Clean Public Web3 Application (Arbitrum Sepolia ETH & Paxos USDG): https://orb-azure-ten.vercel.app/

When a user uploads an image or video across Orb's core modes (Decode, Enhance, or Motion/Video Analysis), Orb executes a three-stage Agentic Vision Workflow:

  1. Deterministic OpenCV 5.0 Perception: Extracts Canny edge density (cv2.Canny), probabilistic Hough structural lines (cv2.HoughLinesP), Laplacian focus variance (cv2.Laplacian), CIE LAB luminance/exposure telemetry alongside RGB dominant chromatic clusters (cv2.kmeans), 3x3 rule-of-thirds spatial energy grids, primary contour saliency bounding boxes, and dense Gunnar Farnebäck optical flow vectors (cv2.calcOpticalFlowFarneback) for video streams.

  2. Autonomous Decision Routing & Self-Healing Vision: Evaluates whether incoming visual evidence is degraded (specifically checking if mean_luminance < 60.0 or contrast_std < 32.0). When triggered, the agent dynamically alters its execution path invoking an OpenCV 5.0 LAB-space CLAHE (cv2.createCLAHE(clipLimit=2.5, tileGridSize=(8,8))) shadow-recovery action on the L-channel, recomputing post-recovery Canny edge-density and Laplacian sharpness gains, and routing the remediated frame alongside structured OpenCV telemetry into Gemini for grounded multimodal synthesis.

  3. Interactive OpenCV 5.0 Vision HUD & Cloud Artifact Streaming: Users toggle directly in the browser between Original Reference, the synthesized OpenCV Vision HUD (displaying neon Canny contours, Hough structural vectors, rule-of-thirds gridlines, and focal RoI crosshairs), and the CLAHE Shadow Recovery output streamed via 24-hour pre-signed URLs from Amazon S3 (s3://orb-opencv5-vision-2026).

How we built it

1. System Architecture

+-----------------------------------------------------------------------------------+ | CLIENT TIER (Vercel / Browser UI) | | - Interactive OpenCV 5.0 Vision HUD Switcher (Original | Vision HUD | CLAHE) | | - Web3 Wallet Auth (EIP-191 Stateless HMAC Nonces + Arbitrum Sepolia 421614) | +-----------------------------------------+-----------------------------------------+ | HTTPS REST (Mangum ASGI) v +-----------------------------------------------------------------------------------+ | AWS COMPUTE TIER: AWS Lambda Graviton2 (arm64 / aarch64 NEON) | | Endpoint: jsnwiqkw5sqs3c7hag6ywd3ngm0vljpc.lambda-url.us-east-1.on.aws | | | | [1. PERCEPTION: OpenCV 5.0.0.93 Engine] | | ├── Canny Edge Density & 3x3 Spatial Grid Complexity (cv2.Canny) | | ├── Probabilistic Hough Structural Vector Extraction (cv2.HoughLinesP) | | ├── Laplacian Variance Focus/Sharpness Scoring (cv2.Laplacian) | | ├── CIE LAB Luminance Telemetry & RGB K-Means Color Clustering (cv2.kmeans) | | └── Dense Gunnar Farneback Optical Flow for Video (cv2.calcOpticalFlowFarneback)| | | | | v | | [2. AGENTIC DECISION ROUTER] | | └── Evaluates Mean Luminance (< 60.0) & Contrast (< 32.0) -> Branches Execution | | | | | v | | [3. ACTION: Autonomous Remediation + HUD Synthesis + Grounded VLM Inference] | | ├── LAB L-Channel Adaptive Histogram Equalization (cv2.createCLAHE) | | ├── Post-CLAHE Edge Density & Laplacian Sharpness Gain Self-Evaluation | | ├── Multi-Layer Vision HUD Rendering (opencv_hud_overlay.jpg) | | └── Grounded Multimodal Prompt Synthesis (Recovered Frame + OpenCV Telemetry) | +-----------------------------------------+-----------------------------------------+ | boto3 S3 Sync & Atomic Conditional Writes v +-----------------------------------------------------------------------------------+ | AWS STORAGE TIER: Amazon S3 (s3://orb-opencv5-vision-2026) | | - /orb-jobs/<job_id>/opencv_hud_overlay.jpg (24h Pre-Signed URL Delivery) | | - /orb-jobs/<job_id>/opencv_clahe_enhanced.jpg & opencv_telemetry.json | | - /orb-jobs/<job_id>/job_state.json (Cross-Container Serverless Job State) | | - /orb-credits/ (Encrypted Atomic Wallet Ledger & Tx Replay Protection) | +-----------------------------------------------------------------------------------+

Challenges we ran into

Overcoming Cloud Quota Constraints by Re-Architecting for Serverless AWS Graviton2 (arm64): New AWS accounts face strict initial EC2 vCPU limits that block standard multi-vCPU instance launches. Rather than settling for off-AWS compute or a slow sleep-prone host, we re-architected our entire FastAPI + OpenCV 5.0 stack into a native linux/arm64 serverless deployment on AWS Lambda Graviton2 backed by Amazon S3 (s3://orb-opencv5-vision-2026).

Stateless Cross-Container Persistence Across Warm Lambda Instances: Because serverless Lambda containers do not share local /tmp storage, background job polling, cryptographic wallet challenges, and credit balances would normally fragment across concurrent warm instances. We engineered synchronous serverless job execution paired with S3 state fallback (job_state.json), stateless HMAC-SHA256 challenge/session tokens (sc. and ws_), and an encrypted S3 ledger with conditional atomic writes.

Web3 Wallet Extension Simulation Quirks on Arbitrum Sepolia (421614): Browser wallet extensions like Zerion return internal 404 errors when their proprietary simulation indexers fail to recognize Arbitrum Sepolia testnet transactions. We solved this by implementing automatic EIP-3085 chain/RPC registration (https://sepolia-rollup.arbitrum.io/rpc), explicit gas floors (>= 100,000 or live estimate + 20%), and graceful recovery paths while keeping the clean public URL (/) strictly separated from the judge evaluation link (/?judge=OPENCV2026).

Known Limitations & Failure Cases:

0-Bit Sensor Clipping (Luminance < 5 or > 250): When shadow regions contain pure zeroed pixels rather than compressed dark tones, LAB-space CLAHE cannot recover lost sensor data and can amplify JPEG macroblocking.

High-Frequency Background Textures: Dense background textures (such as foliage or gravel) can elevate Canny edge density in peripheral grid cells, occasionally pulling the heuristic saliency bounding box away from smooth foreground subjects.

Serverless Memory Ceiling on High-Bitrate Video: Within the AWS Free Plan 512 MB Lambda memory envelope, ultra-long 4K video streams can approach /tmp and RAM bounds; Orb mitigates this via upload size caps and frame downsampling during Farnebäck optical flow extraction.

Accomplishments that we're proud of

True Closed-Loop Agentic Vision with OpenCV 5.0: Building a system where OpenCV 5.0 doesn't just decorate the UI, but actively measures visual quality (mean_luminance, contrast_std, Laplacian sharpness, edge density), decides when an input is too dark for reliable perception, and autonomously executes LAB-space CLAHE shadow recovery measuring and surfacing live post-recovery gains (+65.0% Canny edge-density gain and +121.8% Laplacian sharpness gain on our real-world corridor scene; +12.5% edge gain and +39.8% sharpness gain on synthetic 1080p underexposed benchmarks) before multimodal reasoning.

Zero-Cold-Sleep Native AWS Graviton2 (arm64) + S3 Production Stack: Delivering verified 1080p OpenCV 5.0 telemetry and autonomous CLAHE recovery in 865.3 ms mean (838.4–900.0 ms) for underexposed scenes and 948.4 ms mean for standard scenes on a 512 MB aarch64 AWS Graviton2 Lambda instance (with sub-260ms local x86_64 baselines), paired with automatic S3 artifact persistence and 24-hour pre-signed URL streaming.

Comprehensive Verification Suite: Reaching 246 passing frontend tests (vitest) and 343 passing backend unit and integration tests (pytest) covering OpenCV 5.0 telemetry, post-CLAHE self-evaluation gains, S3 artifact sync, and stateless cryptographic credit ledgers.

What we learned

Classical CV + VLMs Are Multiplicative, Not Redundant: Conditioning a VLM on both an OpenCV CLAHE-remediated frame and deterministic geometry (Canny edge density, Hough line counts, Laplacian focus scores, CIE LAB luminance, and RGB color clusters) dramatically reduces spatial hallucination and anchors the model's reasoning in verifiable math.

CIE LAB L-Channel Isolation Is Essential for Autonomous Enhancement: Applying histogram equalization directly to RGB channels distorts hue balances and amplifies color noise; isolating OpenCV's cv2.createCLAHE strictly to the lightness (L) channel in CIE LAB space preserves chromatic fidelity while recovering shadow structure.

Designing for Stateless Serverless Compute: Moving stateful vision jobs and cryptographic ledgers onto AWS Lambda taught us how to combine S3 conditional writes with HMAC-signed tokens to achieve full horizontal scalability without database connection-pool bottlenecks.

What's next for Orb

Launching "Create" Mode (Closed-Loop Generative Synthesis Guided by OpenCV 5.0): Building out Orb's dedicated Create mode to turn our analytical telemetry into an active generative studio. Users will be able to take the deterministic primitives extracted during Decode and Enhance such as Canny edge maps, Hough perspective lines, rule-of-thirds saliency masks, and K-Means color palettes and feed them directly into Create mode as structural control layers to generate new images and videos that strictly obey the geometry and lighting of the reference scene.

Self-Correcting Agentic Generation in "Create" Mode: Instead of one-shot generation, Create mode will run every newly synthesized visual through our OpenCV 5.0 perception engine (cv2.Laplacian sharpness, cv2.Canny structural alignment, and LAB histogram balance) before showing it to the user autonomously applying CLAHE tonal balancing or triggering an iterative refinement pass if the generated output misses the target composition metrics.

Real-Time Arm64 Video & ControlNet Pipelines on AWS Graviton: Scaling Create and Decode modes on AWS Graviton processors to support multi-frame optical-flow-guided video synthesis and batch creative asset pipelines persisted directly to Amazon S3.

Built With

Share this project:

Updates

Submission history