SiteSentinel AI

AI-powered construction safety monitoring that detects PPE violations and unsafe job-site activity from video, then generates timestamped evidence reports for faster review.

Comment

Inspiration

Construction sites generate hours of video every day, but most of that footage is only reviewed after something goes wrong. Safety officers do not have time to manually watch every camera feed, and small PPE violations can quickly become serious incidents.

SiteSentinel AI was built to turn raw construction-site footage into actionable safety intelligence. The goal is simple: help safety teams detect PPE violations and unsafe job-site behavior faster, with timestamped evidence they can review immediately.

What it does

SiteSentinel AI analyzes construction-site videos and detects safety-related events such as missing hard hats, missing safety vests, unsafe activity near equipment, restricted-area access, and other visible job-site risks.

For each finding, the system generates structured safety alerts with:

  • Timestamp
  • Camera ID
  • Violation type
  • Severity level
  • Confidence score
  • Evidence summary
  • Exportable report output

The final output helps safety officers quickly review the highest-risk moments instead of manually scanning through long videos.

How we built it

We built SiteSentinel AI using a video-intelligence pipeline powered by TwelveLabs models through AWS Bedrock.

The workflow starts with construction videos stored as S3 URIs. The videos are processed in a SageMaker/Jupyter environment, where Marengo is used to search for relevant safety moments in the footage. Pegasus is then used to reason over those moments and generate structured descriptions of what happened.

The system converts raw video analysis into organized safety findings, ranks them by severity, and exports the results into formats such as CSV, JSON, and an HTML report.

Challenges we faced

The hardest part was making the system useful for real construction footage, not just clean demo clips. Construction videos often have occlusion, bad camera angles, lighting changes, workers partially visible, and cluttered environments.

Another challenge was avoiding an overly broad system. Instead of trying to detect every possible safety issue, we focused on a small set of high-value categories like PPE compliance and visible unsafe behavior. That made the project more practical and reliable within the hackathon timeframe.

What we learned

We learned that video intelligence becomes much more useful when it is tied to operational workflow. A detection alone is not enough. Safety teams need timestamps, severity ranking, evidence, and exportable reports.

We also learned how Marengo and Pegasus can work together: Marengo helps find relevant moments in video, while Pegasus helps describe and structure those moments into useful safety findings.

What’s next

Next, we want to improve SiteSentinel AI with:

  • A full web dashboard for reviewing safety findings
  • Automatic evidence clip extraction
  • OSHA citation mapping
  • Multi-camera timeline support
  • PDF report export
  • Human-in-the-loop review and correction
  • Larger validation set with more construction scenarios

SiteSentinel AI is designed as a practical safety-review assistant for construction teams, helping reduce manual review time while making job-site risks easier to catch and document.

Built With

Share this project:

Updates