Agentic Lab Vision

• 

  Architecture: OpenCV 5.0 on AWS Lambda behind a public URL; alerts via SNS, decisions logged to CloudWatch. No servers, no database.

• 

  What OpenCV sees decides the next tool call: repair glare or darkness, refuse a blur, or hand a human an estimate to approve.

• 

  Glare photo: repaired highlights (pink) are excluded from the count and reported, here 10% of the dish. The count is a lower bound.

• 

  Dim photo: the agent called enhance, looked again, then counted. Red circles are detected colonies; every count is only an estimate.

Inspiration

Culture labs still count colonies and judge plate photos by eye. I run a Listeria phage-engineering project, Project Phage, where that bottleneck limits how fast we can test candidates.

What it does

Upload a photo of a culture plate. OpenCV 5 measures the photo (dish found, focus, light, glare) and counts colonies. A perception, decision and action loop then chooses what to do: repair glare or darkness and look again, refuse a blurry photo and ask for a retake, escalate a crowded plate, or hand you an estimate to approve. No count is ever recorded automatically.

How I built it

OpenCV 5.0.0 on AWS Lambda (Python 3.13) behind a Function URL, split into two layers because OpenCV 5 exceeds the 50 MB upload limit. Amazon SNS sends alerts, CloudWatch Logs keeps one line per verdict and per human decision. The agent is rule-based, with no language model at runtime. Every step, and the measurements behind it, comes back as a trace. Deployment is scripted and idempotent.

Challenges we ran into

Counting is hard: the colonies are small pale dots on pale agar, and my first counters locked onto the dish rim and printed text. I measured that, built a learned filter, and found it did not clearly beat the simple counter, so I kept the simple one and made the agent honest about it. The stress test also caught my own bug: dimming a photo lowered my sharpness metric, so the agent wrongly asked for retakes; I fixed it by normalising sharpness by contrast.

Accomplishments that we're proud of

Photo triage measured on expert-labelled plates (AGAR demo, 40 plates): blur caught 20/20, dimming handled 18/20, glare handled 20/20, and glare repair brings counting error back to the clean-photo level. The system refuses to count what it cannot trust and shows a person what it hid.

What we learned

Report the weak number. The counting accuracy is low (average error 29 colonies; 6 of 38 plates within 20%), so the design keeps a person in the loop instead of hiding it.

What's next for Agentic Lab Vision

Real Listeria plates on chromogenic agar, a better counter, contamination detection evaluated on enough examples, and the original proposal's camera-based optical density with a calibrated optical bench.

Built With

• agar
• agar-dataset-(cc-by-nc-2.0;-majchrowska-et-al.
• amazon-cloudwatch-logs
• amazon-sns
• aws-lambda
• claude
• numpy
• opencv-5
• python