ShadowFlow

Inspiration

The "Automation Gap." While AI coding agents are making it easier to build software, bridging the gap between non-technical knowledge workers and enterprise RPA bots remains a massive bottleneck. Process discovery usually requires tedious interviews, mapping, and manual code reconstruction by RPA developers. We wanted to eliminate this friction by allowing anyone to turn their everyday manual actions into production-ready automations instantly.

What it does

ShadowFlow is an intelligent observation agent that acts as a front-end to the UiPath platform. It aligns with Track 2: UiPath Maestro BPMN and Track 1: UiPath Maestro Case.

• Observe: It watches knowledge workers execute tasks in real-time (e.g., SAP invoice processing, Excel reporting), captures the underlying UI telemetry, and uses AI to infer semantic intent.
• Crystallize: With one click, it crystallizes these actions into standard, ReFramework-compatible UiPath .xaml workflows—complete with variables, retry scopes, and Try-Catch error boundaries.
• Self-Heal: If a bot's UI selector breaks in production, ShadowFlow triggers a human-in-the-loop exception handling step. It re-observes the screen, patches the broken selector, and seamlessly redeploys the bot.

How we built it

• Frontend: React 19, Vite, Tailwind CSS, TanStack Query for a highly responsive, enterprise-grade dark mode UI.
• Backend: Express 5 API Server, PostgreSQL, Drizzle ORM. We used Server-Sent Events (SSE) to stream live telemetry.
• AI & Compilation: We used OpenAI (GPT-4o) for real-time semantic narration of user actions. Our backend features a custom, deterministic compiler that maps this telemetry directly to UiPath XML schemas (e.g., <ui:TypeInto>, <ui:Click>), ensuring the output is perfectly compatible with UiPath Studio.
• Coding Agents (Bonus): We heavily leveraged coding agents (Gemini CLI and Claude Code) to scaffold our API schema, construct the ReFramework XAML compiler logic, and build our Playwright E2E browser test harness.

Challenges we ran into

• Reliable XAML Generation: Relying purely on LLMs to write complex XML often results in syntax errors or hallucinated UiPath selectors. We solved this by building a deterministic compiler that strictly maps telemetry to UiPath's XML schema, restricting the LLM's role to generating human-readable narration and ROI estimations.
• Live Exception Handling: Ensuring that bots don't just fail silently. We had to architect a real-time SSE loop that allows a bot to pause, alert a human operator, ingest new screen fingerprints, and patch its own XAML selectors on the fly.

Accomplishments that we're proud of

• Successfully generating highly structured, valid UiPath ReFramework .xaml files dynamically from a web interface that can be opened directly in UiPath Studio.
• Achieving a 100% test pass rate with a comprehensive Playwright E2E testing suite, constructed rapidly with the help of AI coding agents.
• Designing a stunning, premium UI dashboard that tracks system-wide financial ROI, reclaimed hours, and active bot telemetry in real-time.

What we learned

• We mastered the intricacies of the UiPath ReFramework structure, learning how to programmatically construct <TryCatch> blocks, retry scopes, and <ui:LogMessage> activities using raw XML compilation.
• We discovered the immense power of pairing AI observation with deterministic compilation—proving that AI agents work best when they augment robust RPA platforms like UiPath Automation Cloud, rather than attempting to replace them entirely.

What's next for ShadowFlow

• Direct Orchestrator API Integration: We plan to integrate directly with the UiPath Automation Cloud API to push our generated XAML packages directly to unattended robot queues without requiring manual downloads.
• Native Desktop Agent: Evolving our web-based simulated observation feed into a native desktop application capable of capturing raw OS-level accessibility trees for even deeper, cross-application UI telemetry.

Built With

• drizzle-orm
• express.js
• node.js
• openai
• playwright
• postgresql
• react
• tailwind-css
• typescript
• uipath
• uipath-maestro
• vite