Inspiration
Every organization has valuable operational knowledge trapped inside people's heads.
An experienced support agent knows how to resolve complicated billing issues. An HR specialist knows how to process employee changes. A finance operator knows when an exception requires approval.
Yet turning that knowledge into automation usually requires developers, workflow designers, or complex rule-based systems.
We asked ourselves a simple question:
What if AI could learn how work gets done simply by watching someone do it?
Instead of forcing humans to explain processes to software, we wanted software to learn directly from human demonstrations.
That's how ShadowOps was born: an AI-powered system that transforms everyday operational knowledge into reusable, executable workflows.
What it does
ShadowOps is an AI-powered workflow learning platform that transforms human demonstrations into reusable, autonomous agents.
Users teach ShadowOps by completing a task inside an operational workspace. The system captures meaningful actions, understands their intent, and generates a structured workflow that can be reviewed and reused.
The experience has four stages:
1. Observe — Learn from human actions
A user enables Teach Mode and performs a business process, such as resolving a duplicate payment. ShadowOps captures semantic actions rather than mouse coordinates or screen recordings.
2. Learn — Convert demonstrations into executable workflows
NVIDIA Nemotron analyzes the demonstration, identifies dependencies and decision points, and generates a structured workflow composed of reusable skills.
3. Generalize — Improve through additional demonstrations
When a second demonstration introduces a variation, ShadowOps can identify potential new conditions. For example, refunds above a certain threshold may require manager approval. The user reviews the suggested rule before updating the workflow.
4. Execute — Complete future tasks autonomously
For similar cases, ShadowOps executes the learned workflow through a controlled skill registry, recording each step and pausing when human approval is required.
The initial application focuses on customer support operations, demonstrating how repeated billing and refund workflows can become reusable organizational knowledge.
Our vision: Turn human operational expertise into an organization's reusable AI skill library.
How we built it
ShadowOps is designed around a separation between AI reasoning and deterministic workflow execution.
Frontend and application layer
- Next.js, React, and TypeScript
- Tailwind CSS and shadcn/ui
- Interactive operational workspace, Teach Mode, workflow visualization, and execution history
AI infrastructure
- NVIDIA Nemotron models served through Nebius Token Factory
- Structured model outputs for workflow induction and refinement
- Model-assisted workflow matching, action interpretation, and customer-response generation
Workflow intelligence
- Semantic event capture records meaningful business operations
- Workflow induction converts demonstrations into a structured intermediate representation (IR)
- Demonstration comparison identifies potential workflow variations and conditional rules
- Schema validation checks generated workflows against supported capabilities
Execution engine
- A typed skill registry exposes supported business operations
- A deterministic workflow executor resolves dependencies, evaluates conditions, and invokes skills
- Approval gates prevent sensitive actions from proceeding without authorization
- Execution traces provide visibility into agent behavior
Data and persistence
- PostgreSQL for demonstrations, workflow definitions, versions, and execution records
- Simulated customer support, payment, policy, and approval services for reproducible testing
The key architectural principle is simple:
AI understands and proposes. Software validates and executes. Humans retain control over sensitive decisions.
This approach makes learned workflows more predictable, auditable, and reusable than directly executing free-form LLM-generated instructions.
Challenges we ran into
Understanding intent instead of replaying clicks
Traditional automation records interactions. ShadowOps needs to understand what those interactions mean. We addressed this by representing human activity as semantic business events rather than mouse coordinates or UI selectors.
Generalizing from limited demonstrations
Two demonstrations may represent the same workflow with different conditions. Distinguishing reusable rules from incidental differences is challenging. Our approach uses structured workflow comparison and explicit human confirmation before accepting newly inferred rules.
Balancing autonomy with safety
An agent capable of executing business operations must respect approval boundaries. We designed a typed skill registry, workflow validation, and approval checkpoints to separate AI-generated reasoning from authorized actions.
Making AI-generated workflows executable
Natural-language instructions are flexible but difficult to execute reliably. We introduced a structured workflow intermediate representation that supports validation, deterministic execution, and versioning.
Keeping the demonstration realistic and reproducible
General-purpose browser automation introduces many external dependencies. We focused on a controlled business workspace to demonstrate the core workflow-learning capability without relying on unstable third-party environments.
Accomplishments that we're proud of
The most important achievement behind ShadowOps is its approach to transforming human demonstrations into structured, reusable operational knowledge.
Key product and engineering capabilities:
- Demonstration-driven workflow creation: Users can teach workflows through actions rather than manually programming them.
- Semantic workflow understanding: The system reasons about operational intent, dependencies, and outcomes.
- Multi-demonstration learning: New examples can reveal conditional branches and improvements to existing workflows.
- Reusable skill architecture: Workflows are composed from validated, reusable capabilities.
- Human-controlled autonomy: Sensitive operations can require explicit approval.
- Transparent execution: Workflow runs can be inspected step by step.
We're particularly proud of the architectural distinction between learning a workflow and executing it. NVIDIA Nemotron provides the reasoning layer, while a structured execution engine turns that reasoning into controlled actions.
ShadowOps is intended to demonstrate a path from AI-assisted work toward operational knowledge that organizations can reuse and improve over time.
What we learned
Building ShadowOps has reinforced an important insight: the hardest part of business automation is often not executing actions, but understanding the knowledge and decisions behind them.
A sequence of clicks is not necessarily a workflow. A useful automation must understand intent, dependencies, exceptions, and approval boundaries.
We also learned the importance of separating probabilistic reasoning from deterministic execution. Large language models are powerful at interpreting demonstrations and proposing abstractions, while structured software systems provide stronger guarantees for validation and execution.
Another key lesson is that autonomous systems should not treat every observed behavior as a universal rule. Additional demonstrations can provide evidence for improvements, but inferred changes should remain reviewable and testable.
Finally, using open models through independent AI infrastructure creates opportunities to develop agentic applications with greater control over model selection, deployment, and system architecture.
What's next for ShadowOps — Turn Human Work into Reusable AI Skills
ShadowOps begins with a controlled operational workspace, but our long-term vision is to make workflow learning available across real business systems.
Next steps include:
- Browser-based observation: Learn workflows across websites through semantic DOM events and browser automation.
- MCP and API integrations: Connect with CRM, HR, finance, customer support, and productivity platforms.
- Organizational skill library: Allow teams to discover, share, version, and reuse learned skills.
- Continuous workflow improvement: Identify repeated exceptions, execution failures, and opportunities to optimize workflows.
- Workflow simulation and evaluation: Test learned workflows against synthetic scenarios before production deployment.
- Enterprise governance: Introduce role-based permissions, audit trails, policy enforcement, and configurable approval rules.
Our ultimate goal is to build an AI operations platform where organizations no longer need to manually translate every business process into automation.
Instead, operational knowledge becomes something AI can observe, understand, execute, and continuously improve.
Built With
- express.js
- localstorage
- nebius-token-factory
- node.js
- nvidia-nemotron
- react
- tailwind-css
- typescript
- vite
- zod
Log in or sign up for Devpost to join the conversation.