Team floorsight

Turn existing CCTV into floor measurements and reviewable forklift warnings. A vision agent checks the evidence before an alert; a simulator demonstrates automatic slowing and stopping.

Comment

Inspiration

Forklifts and people share busy factory floors. A camera can make a safe pass look dangerously close because pixels do not measure distance on the floor. We wanted to make existing CCTV useful for checking risk and give supervisors evidence they can act on.

What it does

Floorsight maps people and forklifts onto a floor plan, estimates distances in metres and predicts close passes. A review agent checks recent movement, requests a close-up and a second measurement, and changes its recommendation when the evidence changes. Missing or conflicting evidence stays uncertain. A supervisor reviews floor warnings before alerts are sent. The live judge demo generates four camera views of a simulated warehouse, processes them with the vision pipeline and applies slow/stop/resume commands to the simulated forklift. Real factory clips are processed offline; we do not claim control of real vehicles.

How we built it

OpenCV 5 handles floor geometry, motion, tracking, risk-zone calculations, privacy filtering and video processing. Separate PyTorch models detect people, vehicles and loads in recorded footage. Python and FastAPI serve the review dashboard. AWS Graviton runs the application with COOL. S3 stores evidence, DynamoDB stores track history, CloudWatch records metrics and SNS supports approved alerts. The public repository includes pinned dependencies, Terraform, bootstrap scripts, tests, architecture diagrams and deployment instructions. The live judge simulator keeps alerts inside the dashboard so testing does not message real recipients.

Agentic Vision

The agent follows an evidence-driven tool workflow: replay, close-up, second measurement and recommendation. It can request more evidence or human review rather than force a decision. In controlled synthetic exit tests, false alert records fell from 8 to 0 while all 8 visible hazards still alerted. All 16 blind-camera cases requested review. In 80 synthetic floor-warning cases, hazardous rejection recommendations fell from 16 to 0 and false actionable recommendations fell by 40%. These are controlled synthetic results, not field-accuracy or reviewer-workload claims.

Best Use of COOL

We ran the OpenCV workload on the COOL Graviton AMI and published build information, stock comparisons and output checks. The historical OpenCV chain reached 1.10x the stock speed, while full perception was 0.97x. Our newer experiments do not establish a full-pipeline speedup. Publishing this boundary makes the benchmark reproducible and shows where detector and orchestration costs still dominate.

Challenges and accomplishments

Perspective, unstable headings, occlusion and inconsistent detector results can all create misleading warnings. We added calibrated floor measurements, smoothing, uncertainty handling and tests for conflicting evidence. In one matched simulation, control reduced in-path contacts from 2 to 0, with a throughput tradeoff: 27 metres travelled instead of 80. The submitted implementation passed 306 tests on Linux and Windows; the later judge-demo loop adds three targeted tests.

What we learned

A convincing warning needs reliable measurements and a clear review path. Faster image operations do not necessarily make the whole application faster. We publish raw evidence, failed experiments and limitations alongside successful results. Recorded reviews used scripted approvals. The demonstrated planner is local; Bedrock account access remains blocked.

What's next

Evaluate independently labelled footage and real supervisor decisions, improve complete-pipeline throughput, and validate camera calibration across more sites. Hardware integration and field safety validation are future work.

Share this project:

Updates

Submission history