HSE Audit Agent

See. Assess. Act. Powered by an AI Safety Agent.

Comment

Inspiration

I am a QHSE engineer, and I know from field experience how much time a proper safety risk assessment can require.

The walkthrough itself is rarely the bottleneck. After the inspection, the QHSE professional still has to review photos and videos, identify hazards, assess their severity, determine the applicable requirements, assign responsibilities, define deadlines, write the report and communicate the corrective actions.

Depending on the size and complexity of the site, this process can take several days or even a full week.

I wanted to change that.

The idea behind HSE Audit Agent is simple: if an inspector can capture the workplace with a few photos or a short video, why should the rest of the process still take days?

What it does

HSE Audit Agent is an AI-powered HSE inspection agent that transforms workplace photos or walkthrough videos into an actionable, human-validated safety assessment.

The inspector uploads the media and does not need to select the sector manually.

The agent:

  1. Recognises the work environment
  2. Selects the appropriate HSE rule catalogue
  3. Detects safety non-conformities
  4. Evaluates their actual severity from visual context
  5. Explains the reasoning behind its severity assessment
  6. Maps findings to the applicable ISO 45001 requirement
  7. Assigns a responsible role
  8. Determines a corrective-action deadline
  9. Escalates critical situations when necessary
  10. Extracts the evidence frame
  11. Generates the PDF report
  12. Prepares personalised action notifications

The process runs asynchronously, so the inspector can close the browser while the agent continues working.

Human validation

Safety decisions should not be blindly automated.

The QHSE professional remains in control and can approve, modify, reject or add findings before anything is officially dispatched.

The original AI assessment is preserved, making every correction auditable.

Only approved findings appear in the final report and reach the responsible people.

AI analyses. Human validates. The agent executes.

Controlled uncertainty

The agent can also say "I don't know."

If it cannot identify the environment with sufficient confidence, it does not force a sector and produce potentially incorrect findings.

Individual low-confidence findings are also flagged "to verify on site" rather than being presented as confirmed non-conformities.

This is essential for a safety application.

How we built it

The system is built as an asynchronous agentic workflow on Google Cloud.

When an inspection is uploaded, the API immediately returns 202 Accepted. The inspection continues in the background while its progress is tracked through explicit processing stages.

The architecture separates the AI analysis, storage, queue, notification and reporting layers behind neutral interfaces.

This means external dependencies can be replaced without changing the rest of the application.

The safety rules are also separated from the application code.

HSE catalogues, responsibility assignments and analysis instructions are maintained as configuration files. Adding a new sector therefore does not require rewriting the core application.

The current implementation contains two complete and tested referentials:

Construction sites and offices.

The agent uses multimodal AI to analyse the visual evidence and return structured findings that the rest of the workflow can process deterministically.

The system then extracts only the evidence frames required for actual findings and removes the original source media after processing.

Challenges we ran into

Understanding context, not just objects

Detecting an object is not enough for a safety assessment.

The same object can represent completely different levels of risk depending on its environment.

A cable on the floor may be a minor housekeeping issue. An exposed live electrical conductor in a wet environment can be critical.

The agent therefore needs to reason about the visible context, not simply match objects to rules.

Preventing false confidence

A safety AI that confidently produces incorrect findings can become dangerous very quickly.

We therefore designed explicit refusal paths for both sector detection and individual findings.

When confidence is insufficient, the system asks for human verification instead of pretending to know.

Making the AI genuinely agentic

We wanted more than an image analysis result.

The system had to continue working after the inspector leaves, maintain inspection state, move through multiple stages, generate evidence, prepare the report and eventually dispatch approved actions.

Preserving human responsibility

The hardest design question was not how to automate more.

It was deciding what should never be automated without human validation.

For workplace safety, the final decision remains with the QHSE professional.

Handling real-world failures

The architecture also had to account for model failures, storage failures, duplicate dispatches and interrupted processing.

The inspection workflow is therefore designed around explicit states, retryable processing and idempotent notifications.

Accomplishments that we're proud of

The biggest accomplishment is that we built a complete HSE workflow rather than a simple AI detector.

From one photo or video, the system can move through:

Evidence → Environment → Risk → Severity → Requirement → Owner → Deadline → Human validation → Report → Action

We are particularly proud of the agent's ability to recognise uncertainty instead of always producing an answer.

We are also proud of the human-in-the-loop design.

The AI can analyse and propose, but the QHSE professional remains the final authority.

Another important accomplishment is the separation between safety knowledge and application code.

The rules are configuration, allowing the system to evolve to additional sectors without redesigning the application.

Most importantly, the project addresses a real operational problem from my own QHSE experience:

compressing a process that can take days into a workflow that can be completed in a single morning.

What we learned

We learned that building a useful AI agent is not just about choosing a powerful model.

The model is only one part of the solution.

The real challenge is designing everything around it:

  • uncertainty handling
  • human validation
  • evidence management
  • auditability
  • asynchronous execution
  • responsibility assignment
  • notifications
  • privacy
  • failure recovery

We also learned that AI confidence must be treated differently from human certainty.

In safety, "I don't know" is sometimes a much better answer than a confident but incorrect finding.

Finally, we learned that the best use of AI in professional safety is not necessarily to replace the expert.

It is to remove the repetitive work that prevents the expert from focusing on the decisions that actually matter.

What's next for HSE Audit Agent

This submission is the first working version of a larger HSE platform.

The next step is to expand the validated rule catalogues to additional environments such as:

  • Industry
  • Healthcare
  • Logistics
  • Hospitality

The priority will remain validation in the field, rather than simply adding more sectors to a diagram.

We also want to extend the system toward continuous safety monitoring, including integrations with fixed cameras and other visual sources where appropriate.

The long-term vision is an HSE AI Agent that can continuously help organisations move from:

Detection → Decision → Corrective Action → Verification → Closure

The goal is not to replace the QHSE professional.

It is to give them an AI agent that handles the time-consuming work around their expertise.

From days to minutes.

Built With

  • cloud-firestore
  • cloud-secret-manager
  • could-run
  • fastapi
  • geminiapi
  • geminimultimodalanalysis
  • google-cloud
  • html/css/javascript
  • json
  • python
  • reportlab
  • yamal-rule-catalogue
Share this project:

Updates

Submission history