HazardLens

An OpenCV 5 agent that finds trip and slip hazards, checks its own evidence, and asks a person before raising an alert.

Comment

Inspiration

Falls are one of the main ways older people get hurt and lose their independence, and indoor falls are often caused by things on the floor: a trailing charger cable, a bag left in a corridor, a spilled drink. Care homes do walkway checks, but they are manual and seldom recorded. I wanted a phone-camera tool that does more than label a picture. It should decide what to do next, and it should know when it cannot trust what it sees.

What it does

A carer photographs a walkway. An agent built on OpenCV 5 tools then:

  1. Checks the photo. It measures exposure, sharpness and glare. Blurry photos get a retake request with specific advice. Dark photos are enhanced.
  2. Finds candidates. It segments anything that isn't floor and classifies it as cable, clutter or spill.
  3. Double-checks. It zooms into each candidate and compares it with the nearby floor to reject shadows.
  4. Acts on what it found. Clutter becomes a routine work order. Cables and spills become an urgent alert that waits for human approval before anything is sent to staff through Amazon SNS.

The UI shows the annotated photo, every tool call with the reason for it, and the approval queue.

How we built it

  • OpenCV 5.0 (opencv-python-headless==5.0.0.93) does all the perception:
    • Lab colour space, with a robust quadratic illumination fit (cv.solve, SVD)
    • a k-means multi-modal floor model
    • connected components, contour shape analysis (minAreaRect, convexHull, distanceTransform) and fitLine to reject floor seams
    • CLAHE, NL-means denoising, Laplacian sharpness
    • for zoomed inspection: Sobel/morphological-gradient edge tests and HoughLinesP
  • Agent: an executor with typed tools, budgets and three guardrails. The planner is either a deterministic policy or Claude via tool use, through the Anthropic API or Amazon Bedrock. If the LLM fails, the rule planner takes over automatically.
  • AWS deployment, defined in CDK v2 (the stack synthesizes; for judging I demo from the same Docker image locally or via live screen-share):
    • ECS Fargate on Graviton/ARM64 behind an ALB
    • S3 for JSON decision traces (encrypted, private, 30-day expiry, never images)
    • SNS for human-approved alerts
    • Secrets Manager for the approver token
    • CloudWatch logs, Container Insights, alarms and metric filters for escalations and guardrail interventions
  • Technical report: docs/TECHNICAL_REPORT.md; architecture and agent-workflow diagrams in docs/; evaluation in eval/RESULTS.md.

Challenges

  • A large spill can look like "background" to a median-filter illumination model. Switching to a robust polynomial fit took spill detection from 2 of 8 to 6 of 8 on the dev set.
  • Telling soft shadows from dark objects needs a local comparison. That is why the agent zooms in instead of tightening one global threshold.
  • Deciding where the LLM's autonomy stops. The agent cannot page a person: it can only request it.

Accomplishments

On 60 synthetic scenes, the agent picks the right final action 95.0% of the time, against 73.3% for a single-shot OpenCV pipeline. On a held-out set the figures are 93.3% vs 71.7%. It removes every shadow false alarm, sends every blurry photo back for a retake, and has zero false escalations. The failure cases (clear liquid on light tiles) are published in the evaluation.

What we learned

The extra value of an agent came from choosing which OpenCV tool to run next: re-detecting at higher sensitivity after low-light enhancement, looking harder for liquid when glare is present, and verifying before acting. Generating language was not where the gains came from.

What's next

  • A consented real-world pilot with a care provider
  • Depth from a second frame, to estimate how high a trip hazard sticks up
  • DynamoDB-backed approvals and Cognito per-user identity
  • A COOL/Graviton benchmark of the OpenCV workload

Built With

Share this project:

Updates

Submission history