Plant Debugger: Agentic Field Partner for Brownfield Ops

Inspiration: The Last Meter Has No API

Having worked in production engineering, my experience has shown that full automation is virtually non-existent in most operating facilities—at best, facilities have retrofitted a few sensors to monitor equipment telemetry. The vast majority of brownfield plants still rely on analog visual gauges and manual bypass valves.

If a plant were truly 100% automated with remote actuators on every single line, it should be operated by deterministic control logic (PLCs/DCS) and advanced mathematical/physics models (such as Model Predictive Control) rather than LLMs.

However, in real-world brownfield plants, while central telemetry can detect that an anomaly occurred, the critical observations and actions needed for troubleshooting—local differential-pressure gauges, physical valve positions, and practical access constraints—remain outside the API boundary. The troubleshooting "last meter" has no API.

Conventional AI assistants either provide generic text suggestions or treat humans as mere "approval button clickers" in front of digital workflows.

Plant Debugger fundamentally changes this dynamic:

Google ADK + Gemini owns the diagnostic objective, dynamically selecting what physical observation or bounded experiment must happen next, pausing (RequestInput -> PAUSE) until reality responds, and resuming (FunctionResponse -> RESUME) to re-plan.
The Field Operator provides physical reach that software cannot obtain from APIs alone: analog visual inspection, safe bounded manual adjustments, final-position verification evidence, and explicit site knowledge that can improve later runs.

Human operators are not a fallback for failing autonomy; they are the essential physical interface connecting agent reasoning to the real world.

Collaboration improves across runs

Plant Debugger's collaboration is not limited to a single incident. Brownfield facilities depend on stable local practices that often live only in experienced operators' heads—for example, which maintenance aisle is safe to use while a pump is running.

An operator can explicitly teach one of these site facts. Plant Debugger stores the assertion in Firestore Site Knowledge, supplies it to future ADK planning contexts, and lets later work orders adapt to that human-authored knowledge. The next incident can therefore start with better operational context than the previous one.

This memory is intentionally conservative: only explicit operator assertions become durable Site Knowledge. Gemini's diagnoses, image interpretations, and temporary incident evidence are never silently promoted into long-term plant facts.

What it does

The representative end-to-end troubleshooting demo scenario begins with an observable symptom:

P-101 LOW-FLOW INCIDENT

The hidden simulator fault label is not supplied to ADK.

A representative evidence-led troubleshooting run proceeds as follows:

Anomaly Detection & Event Delivery: FT-101 reports low flow while pump P-101 remains running. The simulator publishes an event to Google Cloud Pub/Sub (plant-anomaly-events), which pushes with OIDC authentication to /api/pubsub/anomaly, recording a verified alert in the Audit Trail.
Context Evaluation: Google ADK + Gemini evaluates telemetry, current hypotheses, completed field results, available bounded tools, and relevant operator-authored Site Knowledge.
Field Inspection Request: Because strainer F-17 has a local differential-pressure indicator that is not remotely instrumented, ADK determines that visual inspection has high information value and dispatches an inspection task, pausing execution (RequestInput -> PAUSE).
Multimodal Evidence Capture: The operator captures and uploads an image containing F-17 and its local gauge. The image itself provides asset context and physical evidence without unnecessary QR ceremonies.
Vision Interpretation: Gemini Vision evaluates visible evidence and reports DP status (NORMAL, HIGH, or UNCERTAIN).
Dynamic Re-planning: The same resumable ADK workflow resumes (FunctionResponse -> RESUME) and chooses the next bounded experiment.
Bounded Experimentation: If ADK selects manual bypass valve BV-104, it determines the target movement (delta_pct, e.g., 10% probe) dynamically from current evidence and valve position rather than a hardcoded script.
Primary Action Confirmation: The operator performs the manual adjustment, returning execution confirmation and subsequent telemetry as causal evidence.
Reasoning from Incomplete Recovery: If flow recovery is insufficient, ADK receives this result as fresh evidence to re-plan (further opening, alternative checks, or escalation).
Physical State-Driven Verification: When ADK requests opening BV-104 fully to mechanically marked OPEN, the policy requires a final-position confirmation photo.
Visual Completion Evidence: Gemini Vision verifies the physical OPEN position from the photo, immediately completing the task without redundant confirmation buttons.
Deterministic Verification: Finally, the deterministic verifier requires sustained healthy telemetry before marking the incident as RESOLVED.

The route is dynamically driven by incoming evidence:

F-17 local ΔP HIGH
+ routing around F-17 improves the plant response
+ BV-104 visibly OPEN when end-state evidence is required
+ FT-101 sustains recovery
= strong converging evidence for F-17 restriction

Gemini never receives authority to declare RESOLVED on its own; deterministic code alone proves recovery.

Collaborative Architecture: How Human and AI Partner Together

1. Agent-led dynamic loop with deterministic safety boundaries

The troubleshooting route itself runs as a resumable Google ADK Dynamic Workflow. Gemini selects the next field capability from current evidence. The application does not hard-code fixed sequences like camera -> +10% -> full open.

Deterministic code owns the non-negotiable safety boundaries:

Allowed capabilities and action bounds;
Step and action budgets;
Durable state persistence and idempotency;
Evidence integrity and SHA-256 validation;
Final-position evidence requirements for critical physical states;
Deterministic recovery thresholds.

ADK chooses and loops. Deterministic code constrains and executes. The physical world supplies the evidence.

2. Physical collaboration via multimodal field evidence

Plant Debugger integrates diverse field evidence:

Remote telemetry streams;
F-17 local differential-pressure gauge photos;
Manual action execution confirmations;
BV-104 physical valve position verification photos;
Previous incident run history;
Deterministic verifier telemetry analysis;
Operator-authored Site Knowledge.

Rather than reducing human collaboration to a passive approval gate, the operator acts as the physical "eyes and hands" on site, creating true reciprocal collaboration.

3. Human practice improves later runs (Site Knowledge & Memory)

Brownfield facilities hold critical tacit knowledge not found in schematics:

"F-17 is accessed from the west maintenance aisle. East-side access is restricted while P-101 is running."

Plant Debugger allows operators to record explicit human assertions into Firestore Site Knowledge. On future incidents, this is automatically fed into ADK's planning context (FieldPlanningContext.site_knowledge).

When ADK next dispatches an F-17 inspection, it generates a refined, context-aware work order:

"Access F-17 from the west maintenance aisle (East access restricted during P-101 operation).
Return one readable photo of F-17 and its local differential-pressure indicator."

Only explicit human assertions become durable Site Knowledge—preventing AI hallucinations from corrupting plant operational knowledge.

How we built it & Google Cloud Architecture

The architecture separates reasoning, physical authority, durable state, and recovery authority across Google Cloud services:

1. Users & Plant Edge

Commander (Operations Lead): Web browser console displaying live schematics, telemetry trends, ADK decision trace, and unified Audit Trail.
Field Operator: Mobile smartphone UI receiving exact bounded instructions, uploading evidence photos, and submitting Site Knowledge.
Plant Edge (Brownfield): Pump P-101, Strainer F-17, Manual Bypass BV-104, Flow Meter FT-101 with streaming telemetry.

2. Public Application Layer (`plant-debugger` on Cloud Run)

Commander UI / Operator UI: Responsive interfaces for operations lead and field workers.
FastAPI Backend: REST API, evidence ingestion, and deterministic control adapter.
ADK Workflow Host: Google Agent Development Kit hosting the resumable Dynamic Workflow (RequestInput -> PAUSE, FunctionResponse -> RESUME).

3. Private Simulation Layer (`plant-sim` on Cloud Run)

Private Cloud Run service providing safe, repeatable physical dynamics, telemetry generation, and action mirroring. Hidden fault labels remain completely isolated from the ADK planner.

4. AI / Reasoning Layer (Vertex AI — Gemini)

Gemini on Vertex AI:
- Planning: Evaluates current evidence, telemetry, and Site Knowledge to choose the next field capability.
- Multimodal Vision: Strictly classifies gauge states (NORMAL / HIGH / UNCERTAIN) and physical valve positions (OPEN / CLOSED / UNCERTAIN).

5. Data Layer (Google Cloud)

Firestore — Field Ops: Stores incident states, field tasks, Audit Trail, run history, and Site Knowledge.
Firestore — ADK Sessions: Persists ADK workflow checkpoints, events, and pause/resume execution state.
Cloud Storage: Securely stores private original field evidence photos with signed URLs and SHA-256 digests.
Secret Manager: Secure management of credentials and runtime secrets.
Cloud Logging: Real-time aggregation of application, request, and ADK execution logs.

6. Control Boundary

Clear authority split: ADK/Gemini plans next steps, while deterministic code enforces action safety bounds and recovery verification.

Google Cloud hosted configuration

Submission mode runs with fail-closed policies and disabled silent fallbacks:

DECISION_ENGINE=adk
ADK_FALLBACK_POLICY=false
ADK_SESSION_BACKEND=firestore
VISION_ENGINE=gemini
VISION_FALLBACK_POLICY=false
FIELD_STATE_BACKEND=firestore
FIELD_EVIDENCE_BACKEND=gcs

Commander and Operator

Commander

Process schematic and live telemetry;
ADK activity and dynamic execution path;
Decision trace: 1 · NEW PLANT EVIDENCE -> 2 · DIAGNOSIS UPDATE -> 3 · NEXT FIELD TASK;
Learned Site Knowledge repository;
Archived incident history;
Deterministic verification status;
Immutable unified Audit Trail.

Operator

Mobile plant context;
Assigned area, equipment, and target;
Exact bounded instructions;
Inspection image upload;
Partial-action completion reflecting actual ADK-selected delta;
Final-position image verification for fully OPEN states;
Direct Site Knowledge submission.

What is simulated and what is live

Process dynamics and fault injection are simulated in plant-sim for repeatable testing.

The ADK workflow, Gemini planning/vision, pause/resume execution, mobile work orders, Firestore persistence, Cloud Storage evidence pipeline, Site Knowledge, ActionGuard boundaries, and deterministic recovery verification run on the live production application path.

Challenges

We deliberately avoided three common design pitfalls:

Assuming every brownfield asset is remotely controllable;
Reducing human collaboration to a meaningless approval button;
Wrapping a static hardcoded state machine with LLM calls.

We also strictly separated reasoning authority from operational and safety authority to ensure reliable industrial behavior.

Accomplishments

Built the diagnostic loop as a resumable Google ADK Dynamic Workflow.
Implemented true pause/resume across physical field tasks.
Created evidence-driven dynamic planning rather than scripted action counts.
Separated inspection evidence from execution evidence.
Verified physical valve OPEN positions automatically via Gemini Vision.
Propagated human-authored Site Knowledge into future ADK planning contexts.
Enforced deterministic telemetry verification as the sole authority for RESOLVED.
Deployed a fully integrated, verifiable Google Cloud architecture (Cloud Run, Firestore, GCS, Vertex AI).

What's next

As Site Knowledge grows beyond a small set of explicit operator assertions, Plant Debugger could evolve it into a provenance-aware retrieval layer spanning approved SOPs, equipment manuals, past incident evidence, CMMS history, and human-authored site practices.

Rather than placing every stored fact into the agent context, future versions could use asset- and task-aware hybrid retrieval (semantic search plus structured metadata filters) to inject only the most relevant, authorized, and current knowledge into each ADK planning step, while preserving source citations, approval status, and knowledge expiry.

Potential areas for further exploration include incorporating acoustic or thermal field observations, exploring Permit-to-Work / LOTO safety boundaries, and integrating with broader asset management data.