VisionLoop: Autonomous CV5 Agent

Developer agent closing the loop for OpenCV 5: auto-tunes vision parameters on CPU and AWS with grounded F1 evidence and zero guesswork.

Comment

1. General Info

Project Name (Max 40 Characters)

VisionLoop: OpenCV 5 Tuning Agent

(Length: 34 characters)

Elevator Pitch (Max 150 Characters)

Autonomous developer agent that iteratively tests, evaluates, and tunes OpenCV 5 CPU workloads on AWS with grounded visual feedback.

(Length: 127 characters)

2. Sponsor / Special Prizes

Dropdown Selection

  • Selected Prize: Agentic Vision Award
    (Do NOT select "Best Use of COOL Award" as our implementation uses standard CPU-optimized OpenCV 5).

Special Award Consideration Text

VisionLoop is submitted for the Agentic Vision Award as an autonomous developer agent closing the perception-action loop for OpenCV 5:

1. Closed Feedback Loop: OpenCV 5 visual outputs (Precision, Recall, Edge F1, or continuity/noise proxies) and execution timings dynamically determine subsequent parameter proposals rather than following a predetermined sweep.
2. Grounded Guardrails: The agent strictly rejects configurations that compromise output quality (Q < Q_min) regardless of speedup, with explicit failure handling when guardrails are unmet.
3. Observability & Human Control: Provides a real-time UI with an agent reasoning stream, Quality vs. Latency trade-off chart, interactive history table row inspection, and a "↺ Winner" restore control.
4. AWS Component: Bounded serverless container execution on AWS Lambda (Dockerfile.aws) with hard spending safeguards (<1MB payload cap, 15s timeout).

3. Built With (Tags)

Paste the following comma-separated tags into the Built with field:

python, opencv, opencv5, aws, aws-lambda, amazon-ecr, docker, fastapi, uvicorn, pydantic, numpy, scipy, scikit-image, javascript, html5, css3, boto3, pytest, computer-vision, ai-agent, hyperparameter-optimization

4. "Try it out" Links

  • Title / Label: GitHub Repository
  • URL: text https://github.com/DhruvPatel0110/VisionLoop-OpenCV-Hktn

5. Testing Instructions

Paste the following markdown into the Testing instructions field:

### Automated Tests & Interactive Dashboard Reproduction

#### 1. Clone & Environment Setup (Python 3.10+)
```bash
git clone https://github.com/DhruvPatel0110/VisionLoop-OpenCV-Hktn.git
cd VisionLoop-OpenCV-Hktn
pip install -r requirements.txt

2. Run Automated Pytest Suite (38 Passing Tests)

python -m pytest tests/

Executes all unit, integration, vision sanity, evaluator tolerance, agent coordinator, AWS bridge, and UI consistency tests.

3. Launch Interactive Workstation Dashboard

python src/web/server.py

Open your browser and navigate to: http://localhost:8000

4. Features to Verify in the Dashboard:

  1. Benchmark Datasets: Click + New Experiment to test any of the 5 curated benchmark presets (geometric, soft gradient, textured noise, low-contrast, fine detail).
  2. Quality Guardrail Enforcement: Select sample_03_noisy, set the quality slider to 0.95, and launch. Notice how the agent honestly declares NO VALID CANDIDATE when guardrails are unmet rather than fabricating a fake winner.
  3. Interactive History Table: Click any row in the Experiment history table. Panel 3 instantly loads that iteration's output with an INSPECTING badge. Click ↺ Winner in the panel header to restore the selected winner.
  4. Logarithmic Quality vs. Latency Chart: Hover over data points to inspect latency, score, parameters, and status; observe the cyan halo ring highlighting the true incumbent winner.
  5. Reproducibility Audit Export: Click Download Full Trace (.json) or Download Full Report (.md) under the Decision Audit Trail tab. ```

6. Project Story ("About the project")

Paste the following markdown into the About the project field:

## Inspiration

Every computer vision engineer is familiar with the tedious cycle of manual hyperparameter tuning. Finding the right Canny hysteresis thresholds, Sobel aperture sizes, Gaussian blur kernels, and gradient norms across varying illumination, contrast, and noise conditions is traditionally manual, visual, and subjective. When cameras move, lighting changes, or hardware constraints tighten, developers are forced to repeat trial-and-error adjustments.

More critically, conventional benchmarking is often ungrounded: comparing a candidate edge detector to an arbitrary baseline output does not prove correctness. Furthermore, speeding up a vision pipeline is counterproductive if boundary recall collapses. 

We built **VisionLoop** to solve this core developer bottleneck. VisionLoop is an autonomous developer agent that closes the perception-action loop for **OpenCV 5** on CPU and **AWS cloud compute**. It tests, measures, diagnoses, and adapts OpenCV pipelines using grounded visual evidence, strict quality guardrails, and an immutable decision audit trail.

---

## What it does

VisionLoop is an autonomous AI engineering workstation and optimization agent:

1. **Precision Baseline Benchmarking:** Priming CPU caches with 3 warmup passes and taking the median of 5 timed runs ($T_{\text{median}}$) using high-resolution monotonic clocks (`time.perf_counter_ns()`) to eliminate scheduler noise.
2. **Grounded Dual-Mode Quality Evaluation:**
   - **Ground-Truth Mode:** Evaluates calibrated benchmark images against verified reference edge maps using morphological tolerance dilation ($\text{Dilate}(E_{\text{ref}}, 1\text{px})$) per standard BSDS benchmark practices, calculating true Precision, Recall, and Edge $F_1$.
   - **Unlabelled / Proxy Mode:** For unlabelled custom image uploads, computes structural heuristics (edge continuity ratio, isolated single-pixel noise penalties, and density fitness). It explicitly labels scores as *Proxy Quality* rather than fabricating ground truth.
3. **Autonomous Perception-Action Feedback Loop:** The agent evaluates output defects and mutates parameters iteratively:
   - *Over-Suppression Detected:* Drastically lowers hysteresis thresholds to recover subtle low-contrast boundaries.
   - *Noise Artifacts Detected:* Increases thresholds or applies pre-smoothing.
   - *Latency Squeeze:* Once quality criteria are met, bypasses Gaussian filtering ($k_{\text{blur}} = 1$) to eliminate CPU convolution overhead.
   - *Gradient Approximation:* Switches to $L_1$ norm gradient calculation ($\|G_x\| + \|G_y\|$) to replace floating-point Euclidean square-root operations with fast integer math.
4. **Strict Quality Guardrail Enforcement:** Enforces $Q_{\text{candidate}} \ge Q_{\text{min}}$. Any candidate that compromises quality is discarded regardless of speedup. If no candidate meets the threshold, the system transparently reports `NO VALID CANDIDATE`.
5. **Serverless AWS Cloud Bridge:** Packages the OpenCV 5 workload in an AWS Lambda container (`Dockerfile.aws`) with strict spending limits (<1MB payload, 15s execution timeout, and automated teardown).
6. **AI Engineering Dashboard:** Features a 4-panel visual comparison workspace (Input Image, Baseline Output, Best Validated Candidate, Ground Truth Reference), live agent reasoning stream, mathematically aligned logarithmic Quality vs. Latency scatter plot, and one-click JSON/Markdown audit trail exports.

---

## How we built it

VisionLoop is engineered across five modular layers:

- **OpenCV 5 CPU Workload (`src/vision/edge_detector.py`):** Implements a 5-dimensional bounded parameter space across thresholds, aperture kernels, Gaussian smoothing, and gradient norms on verified OpenCV 5.0.0 headless builds.
- **Agent Coordinator & Decision Engine (`src/agent/`):** A finite state machine coordinator (`coordinator.py`) managing the perception-decision-action cycle, coupled with an adaptive heuristic optimizer (`optimizer.py`) formulating parameter mutations based on visual residuals.
- **Grounded Evaluation Harness (`src/eval/evaluator.py`):** Morphological dilation tolerance engine computing precision, recall, and $F_1$, with proxy continuity and noise penalties for unlabelled inputs.
- **AWS Cloud Infrastructure (`src/aws/`):** AWS Lambda container image specification (`Dockerfile.aws`), stateless handler (`lambda_function.py`), unified client bridge (`cloud_executor.py`), automated deployment script (`deploy_lambda.py`), and cost teardown script (`teardown_aws.py`).
- **Interactive Workstation Dashboard (`web/` & `src/web/server.py`):** FastAPI backend serving a responsive, dark cinematic UI (`#050B14`, `#07111F`) crafted in Vanilla HTML5, modern CSS3, and JavaScript, consuming under 47 MB of RAM.

### Core Formulation:

$$\text{Speedup} = \frac{T_{\text{baseline}}}{T_{\text{candidate}}}, \quad \Delta_{\text{latency}} (\%) = \left( \frac{T_{\text{baseline}} - T_{\text{candidate}}}{T_{\text{baseline}}} \right) \times 100$$

$$\text{Precision} = \frac{|E_{\text{cand}} \cap \text{Dilate}(E_{\text{ref}}, 1\text{px})|}{|E_{\text{cand}}|}, \quad \text{Recall} = \frac{|E_{\text{ref}} \cap \text{Dilate}(E_{\text{cand}}, 1\text{px})|}{|E_{\text{ref}}|}$$

$$F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

---

## Challenges we ran into

1. **Ensuring OpenCV 5 Compatibility Without GUI Bloat:** Building and running OpenCV 5 headless builds in both local CPU environments and AWS Lambda container images without heavy X11/Qt/GTK dependencies.
2. **Eliminating CPU Measurement Jitter:** Real-world CPU thermal throttling and OS scheduler interference initially created noisy latency samples. We solved this with a strict 3-warmup + 5-timed-pass median harness.
3. **Scientific Truthfulness vs. Cosmetic Illusions:** Avoiding false "100% F1" claims on unlabelled images. We architected explicit dual evaluation modes so unlabelled uploads never show fabricated references or label heuristics as ground-truth F1.
4. **Logarithmic Chart Coordinate Mapping:** Mathematically mapping the scatter plot's latency axis ($\text{cx} = 45 + \log_{10}(\text{lat}) \times 150$) so points align with logarithmic SVG grid ticks ($1, 10, 100, 1000\text{ ms}$).

---

## Accomplishments that we're proud of

- **Statistically Significant Speedups:** Achieved **$1.4\times$ to $1.5\times$ latency reductions** on CPU workloads while maintaining or improving baseline $F_1$ scores across diverse benchmark datasets.
- **Genuine Agentic Behavior:** Built an agent that dynamically adapts based on visual output rather than executing a pre-baked sweep.
- **38 Passing Automated Tests:** Full test suite covering edge detectors, evaluation math, agent state transitions, AWS bridges, and UI data contracts.
- **Sub-50 MB RAM Footprint:** The entire agent and web dashboard consume under 47 MB of RAM during full 5-pass optimization runs.

---

## What we learned

- High-precision benchmarking on modern multi-core CPUs requires cache priming to obtain statistically significant timing deltas.
- Morphological tolerance dilation is critical for grounded edge evaluation; raw pixel-for-pixel intersection severely penalizes valid 1-pixel boundary offsets.
- Transparent failure handling (clearly showing when guardrails cannot be met) is essential for trustworthy AI engineering tools.

---

## What's next for VisionLoop

- Expanding the parameter search space to adaptive bilateral filtering and multi-scale morphological pyramids.
- Adding automated deployment to AWS Graviton (ARM64) instances for high-throughput edge inference benchmarking.
- Supporting custom multi-stage vision pipelines beyond edge detection, including feature extraction and contour segmentation.

Built With

Share this project:

Updates

Submission history