Agentic Vision Pipeline Repair (AVPR)

• 

Inspiration

Computer vision pipelines can fail even when the underlying algorithm is sound. A small change in an image or a poorly chosen parameter can lead to false positives, missed detections, and unreliable results. Fixing these failures often requires manual experimentation, repeated evaluations, and considerable debugging.

We wanted to explore a different approach: what if a vision pipeline could diagnose its own errors, propose targeted repairs, test those changes, and learn from the evaluation feedback?

That idea became Agentic Vision Pipeline Repair (AVPR), a project focused on making computer vision pipelines more adaptive, measurable, and easier to debug.

What it does

AVPR is an experimental agentic system built around OpenCV 5 that evaluates a vision pipeline and attempts to improve its detection quality through a bounded repair loop.

Its main capabilities include:

• Vision evaluation: Measures detection performance against supplied ground-truth annotations.
• Error diagnosis: Examines false-positive and false-negative patterns to identify potential failure modes.
• Targeted repair proposals: Uses diagnosis-driven heuristics to prioritize parameter configurations suited to the observed errors.
• Experimentation and evaluation: Tests candidate configurations and compares their performance against the current best configuration.
• Acceptance and rollback: Retains improvements when they satisfy the selection criteria and rejects unsuccessful candidates.
• Experiment traceability: Records diagnoses, proposed actions, parameter changes, before-and-after metrics, and acceptance or rejection decisions.

The goal is not simply to produce detections. It is to make the process of diagnosing and improving a vision pipeline explicit and measurable.

How we built it

We built AVPR using Python, OpenCV 5, NumPy, and Streamlit.

The application combines a vision-processing pipeline, an evaluation layer, and a repair agent. The vision pipeline generates detections, while the evaluation layer compares those detections against ground-truth annotations. The repair agent uses the resulting error patterns to prioritize candidate parameter changes.

The workflow follows a feedback loop:

1. Observe: Run the current vision pipeline.
2. Diagnose: Analyze false positives and false negatives.
3. Decide: Prioritize candidate parameter changes based on the diagnosis.
4. Act: Apply a candidate configuration and rerun the pipeline.
5. Evaluate: Measure the resulting detection quality.
6. Keep or roll back: Retain the best-performing configuration rather than blindly accepting every change.

We also implemented a real-image evaluation workflow that accepts uploaded images and corresponding JSON ground-truth circle annotations, allowing users to evaluate the detector against their own labeled examples.

To make the project reproducible, we pinned dependencies, added automated tests, provided container support, and documented setup and execution instructions.

Challenges we ran into

Designing meaningful agentic behavior. We wanted the repair agent to do more than search parameters blindly. We introduced diagnosis-driven proposal ordering so that false-negative-heavy cases prioritize recall-oriented changes, false-positive-heavy cases prioritize precision-oriented changes, and mixed cases prioritize balanced proposals.

Avoiding regressions. A candidate configuration can improve one metric while making another worse. We therefore track evaluation results and preserve the best accepted configuration instead of assuming that every experiment is an improvement.

Evaluating the agent fairly. A higher score than the baseline does not automatically prove that the agent is better than a simpler search strategy. We compared adaptive repair against both a baseline and random search, and retained the trial-level results rather than relying on a single favorable run.

Separating demonstration results from real-world evidence. Synthetic benchmark results are useful for testing the repair mechanism, but they do not establish performance across diverse real-world images. We designed the real-image evaluation path while keeping this limitation explicit.

Accomplishments that we're proud of

• Built a working OpenCV 5 application with an interactive Streamlit interface.
• Implemented a diagnosis-driven repair loop with candidate evaluation and rollback.
• Added a traceable record of the agent's observations, decisions, actions, and outcomes.
• Created a real-image evaluation workflow using user-supplied ground-truth annotations.
• Added automated tests and container-based execution support.
• Established a reproducible benchmark comparing the baseline, adaptive repair, and random search.

In our current five-trial synthetic benchmark, the mean F1 scores were:

Method	Mean F1
Baseline	61.9%
Adaptive repair	69.9%
Random search	67.9%

Adaptive repair improved mean F1 by approximately 8.0 percentage points over the baseline and 2.0 percentage points over random search in this benchmark. However, the estimated 95% confidence interval for the adaptive-versus-random difference included zero, so the results do not establish a statistically significant advantage over random search.

We consider that transparency an important part of the project: meaningful progress includes understanding what the evidence does and does not demonstrate.

What we learned

Building AVPR reinforced that an agentic system needs more than an LLM-style explanation or a sequence of tool calls. Its decisions must affect subsequent actions, and those actions must produce measurable outcomes.

We learned the importance of:

• Connecting diagnoses to concrete, testable actions.
• Using objective metrics to evaluate proposed changes.
• Preserving previous good configurations when experiments fail.
• Comparing adaptive approaches against simple baselines.
• Distinguishing synthetic validation from real-world generalization.
• Making system behavior observable so that users can understand why a repair was attempted and whether it worked.

Most importantly, we learned that a trustworthy repair agent should be able to show its work, including unsuccessful experiments.

What's next for Agentic Vision Pipeline Repair (AVPR)

Our next steps are to expand evaluation across more diverse, labeled real-world images and more challenging vision pipelines. We want to investigate additional diagnosis strategies, broader repair actions, and stronger comparisons against conventional optimization methods.

We also plan to improve the experiment dashboard, add more robust regression safeguards, and explore how the same repair framework could support other OpenCV-based tasks.

Our longer-term vision is a reusable system that helps developers diagnose, repair, and validate computer vision pipelines with less manual trial and error.

AVPR is an initial step toward vision systems that do not just process images, but systematically use evaluation feedback to improve their own pipelines.

Built With

• agentic-ai
• artificial-intelligence
• computer-vision
• docker
• machine-learning
• numpy
• opencv-5
• pytest
• python
• streamlit