PatrolAI

Agentic Physical AI that turns uncertain industrial observations into verified, bounded actions.

Comment

PatrolAI is an industrial inspection agent that chooses to inspect again when the evidence is uncertain instead of guessing.

Inspiration

Industrial inspection systems can fail in two dangerous ways: missing a real hazard or acting confidently on incomplete evidence.

Incomplete evidence can be very ordinary: a camera view may be partially blocked, lighting may be poor, glare may obscure an indicator, or the inspector may simply be looking from the wrong angle.

PatrolAI explores a Physical AI workflow where uncertainty becomes an action.

Instead of forcing an immediate safe-or-dangerous decision, the agent can request REOBSERVE, move to another inspection position, collect new evidence, verify it, and only then allow a bounded action.

The goal is not to replace deterministic safety logic with an LLM. PatrolAI combines deterministic boundaries, contextual AI reasoning, and active re-observation so an autonomous inspector can recognize when it needs more evidence before acting.

What it does

PatrolAI follows a closed-loop workflow:

Sense → Perceive → Reason → Decide → Act → Re-observe when needed

Unity provides the simulated industrial environment and embodied mobile inspector. OpenCV processes visual evidence, and PatrolAI converts that evidence into a structured observation containing the detected condition and evidence quality.

The observation is sent at runtime through Nebius Token Factory to NVIDIA nvidia/Nemotron-3_5-Lightning.

Nemotron returns a constrained structured decision. PatrolAI applies strict parsing, validation, and a deterministic safety envelope before allowing the result to affect the simulated agent.

The agent can take bounded actions such as:

  • CONTINUE
  • REOBSERVE
  • STOP

When evidence is insufficient, PatrolAI can return REOBSERVE. The simulated inspector moves to collect another observation, new evidence is verified, and the reasoning loop runs again before a final action is allowed.

The demo shows three paths:

Scenario A — Clear safe evidence

SAFE → CONTINUE → CONTINUING

Scenario B — Insufficient evidence

UNKNOWN → REOBSERVE → active inspection → new observation → verification → DANGER → STOP → STOPPED

Scenario C — Clear hazard evidence

DANGER → STOP → STOPPED

The interface exposes provider, model, runtime state, evidence, decision, action, confidence, latency, and final state so the reasoning-to-action path remains observable.

How we built it

PatrolAI combines several layers into one bounded Physical AI workflow:

  • Unity 6000.6.3f1 — industrial simulation, embodied inspector, movement, and agent behavior
  • OpenCV + Python — perception and evidence processing
  • Nebius Token Factory — authenticated runtime inference
  • NVIDIA nvidia/Nemotron-3_5-Lightning — structured contextual reasoning
  • C# observation and decision contracts — explicit boundaries between perception, reasoning, and action
  • Strict parsing and policy validation — prevents malformed or inconsistent model output from directly controlling the agent
  • Deterministic safety envelope — constrains final execution
  • Agentic orchestration — maps decisions to CONTINUE, REOBSERVE, and STOP
  • Runtime-truthful evidence UI — distinguishes REAL, MOCK, FALLBACK, FAILED, LOCAL SAMPLE, and NOT VERIFIED

The Windows standalone workflow supports an executable-relative Python runtime with pinned OpenCV and NumPy dependencies. The Nebius API key remains external and is never bundled into the application.

Significant updates during the hackathon

PatrolAI builds on a pre-existing industrial inspection baseline containing the Unity environment, OpenCV perception, mobile-inspector movement, active re-observation movement, fusion/temporal workflows, and existing runtime visualization.

Those inherited capabilities are not presented as new hackathon work.

During the Nebius × NVIDIA hackathon, we added:

  • The independent PatrolAI architecture and product identity
  • Structured observation and decision contracts
  • Live Nebius Token Factory integration with NVIDIA Nemotron
  • Strict parsing, validation, fallback, and deterministic safety boundaries
  • Reasoning-to-action orchestration for CONTINUE, REOBSERVE, and STOP
  • Competition scenarios and runtime-truthful evidence presentation
  • Comparative evaluation, Windows standalone validation, and reproducibility work

The public repository documents this separation between inherited and hackathon-period work.

Challenges we ran into

One major challenge was obtaining strict structured output reliably from a reasoning model.

During early validation, Nemotron accepted the schema but consumed the available completion budget for reasoning and reached finish_reason=length before emitting the required JSON.

We did not weaken the parser or accept malformed output. Instead, we preserved the strict contract, diagnosed the response behavior, and disabled extended thinking only for this short structured-decision request. The following validation completed with finish_reason=stop and valid structured output.

Another challenge was runtime truthfulness.

A demo can look successful even when a cloud request has failed or fallback data is being displayed. PatrolAI therefore keeps REAL, MOCK, FALLBACK, FAILED, LOCAL SAMPLE, and NOT VERIFIED explicitly separate.

A third challenge was adding new AI reasoning without destabilizing the validated inspection baseline. PatrolAI uses adapters and orchestration that call established workflow entry points instead of duplicating the underlying algorithms.

Accomplishments that we're proud of

  • Live Nebius/Nemotron inference can drive a bounded simulated action.
  • Uncertain evidence can trigger active re-observation instead of forcing a premature decision.
  • Model output is parsed and validated before affecting execution.
  • Deterministic safety logic bounds invalid or unsafe behavior.
  • The observation → reasoning → decision → action path remains visible in the UI.
  • Scenario B demonstrates uncertainty → re-observation → verification → final bounded action.
  • Parser, closed-loop, scenario, Windows standalone, and manual Unity validation all completed successfully.

Evaluation

We created a frozen 16-case synthetic/local held-out pilot from structured inspection observations covering safe, hazardous, insufficient-evidence, and conflicting conditions.

The cases are currently labeled internally by the development team; external label review is pending.

Observed action-correct results were:

  • Bounded Rule v2: 14/16
  • Raw Nemotron: 14/16
  • Nemotron + Safety Envelope: 15/16

The hybrid configuration corrected one additional case out of 16 compared with either baseline. We do not treat that one-case difference as statistically significant or as evidence of universal model superiority.

What we learned

The biggest lesson was that agentic AI is not simply about giving a model more autonomy.

For safety-oriented systems, autonomy also means knowing when not to act.

Explicit uncertainty, structured observations, strict validation, deterministic execution boundaries, fallback behavior, and runtime provenance were just as important as the reasoning model itself.

We also learned that AI-versus-rules evaluation requires care. Small benchmarks can contain cues or structural advantages, so evaluation claims need to stay proportional to the evidence.

Finally, real cloud inference is only one part of a Physical AI system. Perception, reasoning, validation, embodied action, observability, failure handling, packaging, and reproducibility all have to work together.

What's next for PatrolAI

The next step is to connect the same:

structured observation → reasoning → bounded action

architecture to real cameras, industrial sensors, mobile robots, or drones.

Moving beyond simulation would require stronger work in:

  • Real-world perception
  • Hardware control
  • Latency and reliability
  • Cybersecurity
  • Fault handling
  • Human oversight
  • Formal safety analysis
  • Independent validation

We also plan to expand the evaluation with independently reviewed labels, more ambiguous and adversarial observations, larger scenario coverage, additional deterministic and model baselines, and real sensor data where appropriate.

Scope and limitations

PatrolAI is currently a Unity-based research prototype.

RUNTIME: REAL refers only to a successful current Nebius/Nemotron inference request; the industrial environment and embodied inspector remain simulated.

The project does not claim physical hardware deployment, production readiness, safety certification, guaranteed hazard detection, measured accident reduction, or measured ROI.

External label review is currently PENDING.

Built With

Share this project:

Updates

Submission history