FOREMAN — Autonomous Agentic Operations Colleague

Architecture

Agentic FOREMAN — the autonomous operations colleague. A horizontal, multi-agent system that turns any signal into a cited, defensible resolution — perceiving, reasoning, acting, and learning, with humans in control.

Inspiration

Every operation on Earth — solar farms, telecom towers, factory lines, fleets, facilities — runs on the same fragile loop: something breaks, a human notices, and the hardest faults wait for one of the few veterans who actually know what to do.

That veteran is a bottleneck. They triage from memory. They chase parts, crews, weather, and warranty status across different disconnected systems. They make a high-stakes call, fix the symptom, and move on — and the root cause, the one that will cause the next ten failures, is never written down. The knowledge walks out the door at the end of the shift.

We didn't want another dashboard that tells a human there's a problem. We wanted a colleague that resolves it — end-to-end, defensibly, and gets smarter every single time. Not a vertical point-solution for one industry, but a horizontal operations brain that treats every breakdown as the same shape of problem: perceive → reason → act → learn.

What it does

A signal lands on any channel — a short video clip / images / audio and a written report on WhatsApp. From that moment, FOREMAN runs the entire case autonomously:

Perceive — a multimodal perception agent powered by Google Gemini watches the video, reads the imagery, cross-references the human's text, and grounds the finding against the asset's real history.
Investigate — a supervisor orchestrates a dynamic crew of specialist agents. An always-on Diagnosis & Recommendation agent runs every case; on top of it, the supervisor invokes only the specialists this case needs — and can justify the ones it deliberately holds back.
Reason — it traces the fault through a knowledge graph to predict the blast radius: every other asset that shares the same install-crew, the same component batch, the same systemic root — flagged before it fails. Site Access agent uses the MCP Connector for real time weather and temperature updates.
Decide (human gate) — when risk is high, a human holds the authority. FOREMAN places a Two-way voice call, states the case, takes the manager's spoken approval, and logs it to the record.
Resolve — on approval it executes governed actions and closes the loop with one fully-cited audit trail sent to everyone involved — every claim sourced.
Learn — a single human thumbs-up promotes the resolution from a candidate skill to a trusted one and writes the systemic root back into the graph, so the next similar case is solved faster, cited to this one.

How we built it

FOREMAN is built natively on the UiPath agentic platform, with a best-of-breed brain on top.

🎼 The conductor — UiPath Maestro Case. A Maestro Case is the durable orchestrator that runs every stage, suspends at each human gate, and resumes on the decision — surviving restarts, waiting hours for an approval without losing state. This is what makes the autonomy safe: it's a governed process, not a loose script.

🤖 The crew — Coded Agents on LangGraph + low-code UiPath Skills. The specialists are UiPath Coded Agents authored as LangGraph state machines — each a focused reasoner with one job (Diagnosis & Recommendation, Safety, Fleet, Parts, Dispatch, Site-Access). We mixed in low-code UiPath Skills for the deterministic steps. Every agent was assisted by Claude Code during development and shipped to the platform through the UiPath Skills CLI — so the whole crew is versioned, deployable, and reproducible, not vibe-coded one-offs.

🧠 The reasoning layer.

A Neo4j knowledge graph models the estate as a graph — assets, crews, component lots, sites — so a single fault becomes a blast-radius query instead of a guess.
A self-learning skill library that grows on human feedback (candidate → trusted).
UiPath Context Grounding keeps the diagnosis agent honest — root causes are grounded on real SOPs and prior skills, with citations.

🗂️ The system of record — UiPath Data Fabric. Sites, assets, vendors, warranties, and batches live as Data Fabric entities, so every agent reasons over the same governed source of truth instead of scattered spreadsheets.

🔌 The action layer — API Workflows, RPA, and the UiPath MCP connector. Downstream actions run through API Workflows and RPA bots. For anything outside the platform, the UiPath MCP connector lets an agent reach live tools on demand, the Site-Access agent pulls live weather to pick a safe repair window. MCP means any new tool plugs in without rewiring the agents.

📡 Omnichannel + the human gate. WhatsApp for intake, a Twilio voice line for the real escalation call, and email for the cited audit — all carried through, with UiPath Action Center as the human-in-the- loop checkpoint.

🖥️ The control room. A React + TypeScript + Vite control-room UI (event-sourced, with a replay engine) makes the entire autonomous run observable — every stage, every agent, every citation, in real time.

Challenges we ran into

Selective orchestration. The hard part of a multi-agent system isn't running every agent — it's running only the right ones and being able to justify the ones you skip. We made the supervisor's restraint a first-class, auditable decision.
Keeping the agents honest. LLM reasoners want to be confident. Wiring Context Grounding
- the knowledge graph so every root cause is cited — not asserted — was essential to make the output defensible.
A truly durable human gate. Pausing a live case for a phone call, capturing a spoken approval, and resuming the exact case state took careful design across Maestro, the Twilio webhook, and Action Center.
Closing the learning loop in real time. Making a single thumbs-up visibly update the knowledge graph and promote a skill — meant building an honest event pipeline from feedback to graph to skill library.

Accomplishments that we're proud of

A genuinely end-to-end agentic case: signal in, resolved-and-audited case out — no human touching a keyboard except to approve and authorize.
A Outbound voice call with a spoken approval captured on the record — the human gate
Blast-radius prediction on a live graph — the same query lighting up in our UI and in Neo4j.
A self-learning loop you can watch happen: thumbs-up → graph updated → skill promoted.
Built horizontally on the full UiPath agentic stack — Maestro, Coded Agents, Skills CLI, Data Fabric, Action Center, MCP, Context Grounding — wired into one coherent system.

What we learned

Orchestration > more agents. A system that chooses and justifies its agents beats one that fans out blindly. Restraint is intelligence.
Defensibility is the product. In real operations, an answer no one can audit is worthless. Citations and a durable trail are not a feature — they're the whole point.
Humans don't want to be replaced — they want leverage. The right design gives the human authority and the agent the work.
The UiPath platform is a real agentic substrate. Maestro's suspend/resume, the Skills CLI, Data Fabric, and MCP let us compose an agentic system instead of gluing scripts together.

What's next for Agentic FOREMAN

More channels, same brain — email, ServiceNow, sensor streams, and ITSM webhooks feeding the identical perceive→reason→act→learn loop.
Deeper graph reasoning — common-cause and criticality scoring to rank which blast-radius to act on first.
Skill marketplace — trusted skills, learned at one site, safely shared across an estate.
Proactive mode — FOREMAN opening its own cases from graph signals, before a human even reports the fault.
Domain packs — telecom, manufacturing, facilities — proving the horizontal claim, one vertical at a time.