RepairMind Edge

An agentic vision system using OpenCV 5 and AWS to inspect electronics, gather visual and electrical evidence, and decide the next troubleshooting action.

Comment

Inspiration

Electronic troubleshooting is rarely a one-shot visual classification problem. A technician observes a device, identifies what information is missing, performs measurements, compares new evidence with available references, and decides what to inspect next.

RepairMind Edge was inspired by this iterative workflow.

The project explores how Agentic Vision can support electronic and edge-device inspection by combining OpenCV 5 perception, structured evidence, human-provided measurements, and an inspection agent that determines the next useful troubleshooting action.

For the demonstration, I used a real barcode scanner PCB as the inspection target.

What it does

RepairMind Edge implements a multi-step perception-decision-action loop.

Starting from a PCB image, OpenCV 5 localizes the board, normalizes its geometry, and generates deterministic visual regions of interest. These perception results are converted into structured evidence and passed to an inspection agent.

The agent evaluates the available evidence and decides whether another observation or physical measurement is required.

In the real demonstration, OpenCV produces 15 evidence items from the PCB image. Because visual evidence alone cannot establish electrical behavior, the agent returns REQUEST_MEASUREMENT.

A real board-supply measurement of 4.927 V is then provided by the human operator and becomes the 16th evidence item.

The new evidence changes the next agent action to REQUEST_OBSERVATION. Since no expected voltage or tolerance is available, RepairMind requests the missing reference specification instead of classifying the measurement as normal or faulty.

The final inspection report therefore records ADDITIONAL_EVIDENCE_REQUIRED rather than inventing an unsupported hardware diagnosis.

How we built it

RepairMind Edge is implemented in Python 3.12 using OpenCV 5, NumPy, Pydantic, pytest, and AWS services.

The core workflow is:

PCB image → OpenCV 5 → structured evidence → inspection agent → next action → new evidence → updated decision → inspection report

OpenCV 5 performs the visual workload, including PCB localization, geometric normalization, contour-based processing, and deterministic visual ROI proposal generation.

The generated ROIs are generic inspection regions, not claimed component detections or detected defects. Their scores are deterministic ranking scores rather than fault probabilities.

The inspection agent operates on typed evidence and produces explicit actions such as REQUEST_OBSERVATION, REQUEST_MEASUREMENT, and REPORT_FINDING. Physical measurements remain human-controlled.

For cloud persistence, RepairMind uses:

RepairMind Edge → AWS SigV4 → API Gateway → AWS_IAM → Lambda → private S3

The Lambda backend validates inspection-session manifests and generates controlled presigned upload URLs. Selected artifacts are uploaded directly to private S3 instead of sending image payloads through Lambda.

The AWS design also uses Block Public Access, server-side encryption, least-privilege IAM permissions, and CloudWatch logging.

To evaluate the Agentic Vision behavior, I created five controlled orchestration scenarios covering uncertain localization, missing visual evidence, measurement requests, missing electrical references, and safe fallback behavior. All five produced their expected actions with zero unsupported diagnoses.

These results evaluate deterministic orchestration behavior; they do not represent 100% hardware diagnostic accuracy.

The implementation also passes 77 core tests and 23 AWS backend tests, for 100 tests across the two controlled test suites, together with Ruff linting and mypy static type checking.

Challenges we ran into

One of the main challenges was defining a safe boundary between visual evidence and electrical conclusions.

A visually interesting PCB region does not prove that a component is defective. Similarly, a measured voltage cannot safely be interpreted without knowing its expected value or tolerance.

Instead of hiding these uncertainties, RepairMind models them explicitly and requests additional evidence when required.

Another challenge was demonstrating genuine Agentic Vision rather than simply explaining a fixed vision result. The OpenCV output had to influence the agent's next action, and newly acquired evidence had to change the subsequent inspection state.

Cloud architecture introduced another design challenge. Rather than sending large image payloads through Lambda, RepairMind uses authenticated session creation and presigned uploads to private S3, keeping the cloud workflow secure and auditable.

Finally, deterministic behavior was important for reproducibility. Evidence construction, ROI selection, agent decisions, reporting, and evaluation scenarios were designed to be inspectable and repeatable.

Accomplishments that we're proud of

RepairMind Edge demonstrates a complete Agentic Vision loop on real electronic hardware rather than a fixed chatbot response.

The project combines OpenCV 5 perception, structured and traceable evidence, deterministic inspection orchestration, human-in-the-loop electrical measurements, explicit uncertainty handling, reproducible reporting, AWS cloud persistence, and automated evaluation.

A result I am particularly proud of is what the system refuses to do.

After receiving the real 4.927 V measurement, RepairMind does not manufacture a diagnosis. Because the expected voltage and tolerance are unavailable, it explicitly requests the missing reference information.

The project also reached a reproducible engineering baseline with 77 core tests, 23 AWS backend tests, static type checking, linting, deterministic evaluation, technical documentation, and a validated end-to-end AWS publication workflow.

What we learned

Building RepairMind Edge reinforced that useful technical AI systems need more than perception output.

For troubleshooting, a system should distinguish clearly between what was observed, what was measured, what can reasonably be inferred, and what evidence is still missing.

I also learned that Agentic Vision becomes more useful when perception is treated as part of an iterative evidence-gathering process rather than as the final answer.

The project also reinforced the importance of human control. Physical measurements remain human-provided, and RepairMind requests additional information when the available evidence cannot support a conclusion.

Finally, AWS integration was most useful when it supported traceability, reproducibility, secure persistence, and inspection-session management rather than simply moving the entire vision workload to the cloud.

What's next for RepairMind Edge

The current version is a hackathon prototype centered on one real hardware inspection case.

The next steps are to expand the system with richer visual evidence extraction, semantic component understanding, reference-aware electrical reasoning, additional real hardware evaluation cases, and a more interactive human-in-the-loop inspection interface.

The cloud workflow can also be extended with explicit session finalization and server-side verification of uploaded artifacts.

Longer term, the goal is to build an inspection assistant capable of combining visual observations, electrical measurements, technical references, and human expertise while maintaining a clear and auditable evidence trail.

The principle remains simple:

Observe → Decide → Measure → Update.

Built With

Share this project:

Updates

Submission history