ShiftGuard Fleet

An autonomous AI fleet that detects factory drift, investigates causes, chooses policy-governed recovery actions, verifies outcomes, replans after failure, and remembers only proven recoveries.

Comment

Inspiration

Most industrial monitoring systems stop at an alert.

They tell an operator that pressure is falling, temperature is rising, or vibration has crossed a threshold—but the difficult work still remains: gather context, search previous incidents, compare recovery options, decide what is safe to execute, verify whether the intervention worked, and escalate when it did not.

We wanted to explore a harder question:

What would an enterprise agent system look like if it were responsible not only for detecting a problem, but for coordinating a safe, verifiable recovery?

That became ShiftGuard Fleet: an autonomous incident-response system demonstrated through a synthetic factory digital twin.

The core design principle is:

Agentic where judgment is useful. Deterministic where safety matters.

What it does

ShiftGuard is event-driven and browser-independent. During each run, its digital twin produces a continuous telemetry stream that is evaluated by a deterministic detector.

The detector looks for correlated changes across coolant pressure, outlet temperature, vibration, flow, production load, pump leakage, and valve response.

When the evidence crosses configured operating conditions, ShiftGuard automatically creates an incident—without waiting for a user prompt.

A fleet of four bounded agents then collaborates:

  1. Incident Coordinator delegates work and manages progress.
  2. Process Investigator examines telemetry, maintenance context, documentation, and verified historical incidents.
  3. Recovery Planner compares eligible interventions and selects one using structured evidence.
  4. Verification Agent coordinates independent verification and explains the result, while deterministic recovery criteria decide whether the process actually recovered.

The complete recovery loop is:

Detect → investigate → retrieve verified memory → plan → authorize → execute → verify → replan if necessary → remember only verified outcomes.

The Planner can recommend an intervention, but it cannot authorize itself.

A deterministic policy gateway separately validates:

  • agent identity and scope;
  • current incident state and version;
  • machine operating envelopes;
  • risk classification;
  • approval requirements;
  • idempotency;
  • stale-state protection;
  • rollback metadata.

Only after authorization can a governed tool mutate the digital twin.

The main demonstration deliberately includes a failed first intervention. The Planner initially chooses a low-impact, reversible throughput reduction. The digital twin reveals that this action was insufficient.

Deterministic verification rejects the recovery. The failed intervention becomes new evidence, the Coordinator replans, and the Planner selects a different intervention.

After the second action measurably changes telemetry, deterministic recovery criteria confirm that the process has returned to its configured recovery envelope.

Only then does ShiftGuard resolve the incident and create a verified institutional memory.

ShiftGuard also demonstrates safe abstention. When one temperature sensor reports an abnormal value while pressure, vibration, flow, and an independent sensor remain normal, the fleet withholds production action and creates a sensor-calibration work order.

How we built it

ShiftGuard combines Google’s managed agent platform with deterministic application services.

Agent platform

The four-role fleet is implemented with Google ADK and deployed as one honest orchestrator topology to Google Agent Runtime.

It uses:

  • Gemini 3.5 Flash-Lite for bounded investigation and planning;
  • Agent Runtime for managed agent execution;
  • Agent Registry for the deployed fleet representation;
  • Memory Bank for verified incident retrieval and promotion;
  • Agent Identity for scoped runtime identity;
  • Agent Gateway for controlled ingress and egress;
  • Model Armor for fail-closed prompt-injection screening;
  • managed logging and tracing for observability.

The Runtime can access only registered outbound endpoints. Private operations tools require a short-lived, audience-bound identity token.

Deterministic operations layer

A private TypeScript Operations service owns:

  • the causal digital twin;
  • telemetry generation;
  • correlated-drift detection;
  • recovery-option eligibility;
  • policy authorization;
  • simulator mutation;
  • incident transitions;
  • verification criteria;
  • idempotency and audit records.

Gemini receives bounded evidence and qualitative forecasts, but never hidden future telemetry, expected outcomes, or scenario answers.

Application infrastructure

We use:

  • React and Vite for the operations command center;
  • Fastify for the Web/API and Operations services;
  • Cloud Run for public and private services;
  • Firestore for durable operational state;
  • Cloud Tasks for browser-independent asynchronous execution;
  • Pub/Sub for event infrastructure;
  • Server-Sent Events for live UI updates.

The workflow continues independently even when the browser is closed or its live event stream is temporarily interrupted.

Measured telemetry grounding

Autonomous actions execute only against the sandboxed digital twin.

To avoid arbitrary demo telemetry, the healthy baseline and cooling-degradation signature are ratio-calibrated against isolated measured cycles from the UCI Condition Monitoring of Hydraulic Systems dataset.

The dataset contains 2,205 measured hydraulic test-rig cycles with pressure, temperature, vibration, flow, cooling efficiency, cooling power, and component-condition labels.

Measured data calibrates realistic signal direction and scale. Intervention outcomes remain explicitly simulated.

Challenges we ran into

The hardest challenge was deciding where agent autonomy should stop.

Allowing Gemini to directly mutate machinery would make the demo look more autonomous, but it would create an indefensible safety boundary. We instead separated recommendation, authorization, execution, and verification into distinct persisted steps.

A second challenge was proving that the Planner genuinely makes a decision. An early architecture generated a fixed option sequence and allowed the model to explain it.

We replaced this with strict structured Planner output. The selected option is validated again before policy evaluation, and invalid or repeated recommendations fail closed without mutation.

Memory required another important trust boundary. ShiftGuard retrieves previous verified incidents as evidence, but memory cannot authorize an action or override current telemetry. Only independently verified, resolved incidents can enter trusted Memory Bank state.

Managed platform integration also exposed real reliability and networking challenges:

  • regional service constraints;
  • private Cloud Run authentication;
  • governed endpoint registration;
  • streamed agent responses;
  • Model Armor screening;
  • transient model capacity;
  • bounded retry behavior.

We treated failures as release blockers rather than hiding them behind recorded fallbacks.

Accomplishments that we're proud of

  • A real four-role Google ADK agent fleet deployed through Agent Runtime.
  • Honest Agent Registry representation without pretending internal sub-agents are separate managed deployments.
  • Genuine structured Planner decisions rather than hardcoded option order.
  • Deterministic policy authorization separated visibly from Gemini recommendations.
  • A failed first intervention followed by evidence-driven replanning and a different successful action.
  • Independent deterministic verification before an incident may resolve.
  • Real Memory Bank retrieval before investigation and promotion only after verified recovery.
  • A safe-abstention scenario that causes no production mutation.
  • Human approval for high-risk simulated shutdown actions.
  • Fail-closed Model Armor proof: a safe request returned HTTP 200, while a poisoned maintenance instruction was blocked with HTTP 403 before agent execution.
  • Enforced Gateway access where unregistered endpoints are denied.
  • A hosted 10+2 reliability matrix passing 12/12 runs with a p95 execution time of approximately 71.7 seconds.
  • A public, browser-independent live application and a credential-free recorded fallback.
  • A bounded Custom Composite lab where judges can create unfamiliar combinations of physical disturbances without passing expected answers into the agent pipeline.

What we learned

The most important lesson was that autonomy does not mean giving an LLM unlimited control.

A strong enterprise agent system combines different forms of authority:

  • models interpret incomplete evidence and compare alternatives;
  • deterministic code enforces safety and state transitions;
  • tools execute only authorized actions;
  • independent verification decides whether the measured result is acceptable;
  • humans retain control over high-risk decisions.

We also learned that recovery is not the same as action execution. An intervention is only a hypothesis until later telemetry proves that it worked.

Finally, institutional memory should contain verified outcomes—not confident explanations. Otherwise, an agent system can turn one unverified assumption into a permanent organizational belief.

What's next for ShiftGuard Fleet

A real deployment would begin in advisory and work-order mode rather than direct industrial control.

The next steps would be:

  • connectors for CMMS, historian, SCADA, and maintenance-document systems;
  • customer-reviewed digital-twin onboarding;
  • broader machine and process models;
  • configurable operating envelopes and approval policies;
  • shadow-mode evaluation against historical incidents;
  • comparison of recommended actions with actual operator decisions;
  • richer long-term retrieval across verified operational outcomes.

ShiftGuard’s current factory is synthetic by design, but the architecture demonstrates the larger idea:

Enterprise agents should not merely recommend actions. They should operate through governed tools, verify real outcomes, replan when they fail, and remember only what has been proven.

Although ShiftGuard is demonstrated through a factory digital twin, its governed recovery pattern is domain-independent:

Detect drift → delegate investigation → select bounded actions → enforce deterministic policy → verify the outcome → remember only verified results.

The same architecture could support software operations, cloud infrastructure, energy systems, and logistics.

In software operations, for example, it could detect a release regression, investigate logs and code changes, recommend a rollback or mitigation, execute it through governed tools, and independently verify service recovery.

Built With

Share this project:

Updates

Submission history