Inspiration
A camera can capture a scene without understanding it. A detector can identify an object without knowing whether another look would help.
We built AutoVision around a different question: What if an agent could use measured visual evidence to decide what to inspect next?
Instead of stopping at a fixed detection or an AI-generated caption, AutoVision combines OpenCV 5 measurements with a bounded decision loop. The agent can inspect a detected region more closely, enhance a dark frame, or request a focused assessment—and then evaluate the returned evidence before deciding what happens next.
Our goal is to give developers reusable vision capabilities without making them rebuild an entire computer-vision platform.
What it does
AutoVision is a standalone visual-analysis service running on AWS, available through a web interface, REST APIs, and Model Context Protocol tools.
It analyzes images and sampled video using OpenCV 5 for motion measurements, object detection, brightness measurements, image transformations, and previews. Optional Amazon Bedrock models provide advisory scene descriptions and focused assessments.
With enhanced inspection enabled, the workflow becomes:
Perceive → Decide → Inspect → Evaluate → Inspect again or recommend human review
This is not simply a chatbot explaining a completed vision result. The visual evidence influences the agent’s subsequent tool calls and recommendation.
How the agent decides to look deeper
OpenCV first measures the supplied images. The agent receives structured observations, including object regions and frame brightness, along with a restricted catalog of available tools.
It can choose to:
- Inspect a region: Crop an existing OpenCV detection and rerun YOLOX to gather additional evidence.
- Enhance a dark frame: Apply OpenCV CLAHE contrast enhancement and rerun detection.
- Assess a scene: Request a focused scene, package, or visible-activity assessment through a deployment-approved Bedrock model.
- Finish: Return an advisory recommendation about whether a person should review the evidence.
Actual tool results return to the agent before its next decision. A missed detection is not silently interpreted as proof that nothing is there.
Operators can configure different approved models for planning and assessment lenses. The agent chooses the inspection tool and focus; the service resolves the corresponding configured model. This provides a controlled way to explore different model configurations without accepting arbitrary downloads, commands, or model endpoints.
A real evidence-changing example
Our saved live OpenCV 5 and Amazon Nova Lite trace captures a useful disagreement:
- Initial full-frame analysis detected a person.
- The agent selected a crop-and-re-detect tool using the measured region.
- The crop returned no person detection.
- The agent subsequently selected a focused scene assessment.
- The workflow finished with human review recommended.
The crop miss remained visible in the evidence rather than being hidden.
The trace also exposed a limitation: part of the planner’s final package-related explanation was unsupported by the returned detections. We document that issue because a completed agent loop is not the same as a correct conclusion.
How we built it
OpenCV 5 performs substantive image processing: decoding, resizing, grayscale conversion, frame differencing, thresholding, changed-pixel measurement, JPEG previews, YOLOX object detection, crop inspection, and low-light enhancement.
Our YOLOX model comes from OpenCV Zoo and is pinned and checksum-verified. It executes through OpenCV’s DNN module.
Amazon Bedrock provides managed model access for the Decider and optional focused assessments. The inspection loop allows at most two follow-ups and three planner calls within a 35-second loop budget, in addition to initial analysis.
We built the surrounding service with:
- Next.js and React: Web interface and authenticated APIs.
- ECS Fargate: Containerized application and Python vision services without managing servers.
- Application Load Balancer: HTTPS routing and service health checks.
- DynamoDB: Organization-scoped jobs, settings, usage, and results.
- SQS and a dead-letter queue: Background incident processing and failed-delivery handling.
- Secrets Manager and IAM: Managed deployment secrets and scoped AWS permissions.
- CloudWatch and structured traces: Operational diagnostics and inspectable agent steps.
- CloudFormation: Repeatable installation and reviewed infrastructure updates.
Not every analysis request uses every service. Interactive analysis and asynchronous incident processing are separate paths.
Standalone service, real application integration
AutoVision does not require a particular camera vendor or application.
Developers can use its REST APIs in conventional software or expose its MCP tools to an external agent. The same service supports hosted SaaS operation and private installation in a customer’s own AWS account.
Evaluation and what we learned
Our evaluation separates deterministic image measurements, detector behavior, agent orchestration, and live model behavior.
Historical local tests matched generated motion masks at 0%, 5%, 50%, and 100% changed pixels. A 56-frame detector evaluation documented object-presence results and machine-specific latency measurements.
These are not hand-labeled precision/recall scores or production throughput guarantees.
Agent tests cover evidence feedback, consent, quota limits, invalid tool requests, unavailable models, and failure handling. Deterministic planner fixtures test orchestration; the saved live trace provides separate evidence of actual OpenCV and Bedrock execution.
We learned that looking closer does not always improve a result. Cropping can remove context, contrast enhancement cannot recover missing detail, and generative explanations can introduce unsupported statements. Preserving disagreement and uncertainty is therefore part of the product—not merely an error message.
Responsible operation and human control
Enhanced inspection requires explicit AI consent. Tools, models, inputs, execution time, and follow-up usage are bounded.
Failures preserve initial evidence and recommend review. Structured traces expose tool choices and results without exposing private model chain-of-thought.
AutoVision does not infer identity or intent, unlock doors, approve security actions, or replace certified alarms. Its review recommendation is advisory; the consuming application and its users control consequential actions.
The browser video workflow submits bounded image samples rather than the original video. Private jobs retain limited sampled evidence under the documented retention and purge policy.
What’s next
Our next priorities are broader labeled day/night and occlusion evaluations, better measurement of appropriate versus unnecessary review recommendations, and comparisons among approved assessment-model configurations.
We want to make the agent’s next question more useful—not simply make its final answer sound more confident.
Try it
Website: https://autovision.dev
Source, setup, and judge’s field guide: https://github.com/techcto/autovision
Log in or sign up for Devpost to join the conversation.