FlowPilot — Autonomous Document Ops

FlowPilot: An Agentic RPA Copilot — Project Story

INSPIRATION

When we started thinking about RPA automation, we noticed a persistent problem: most RPA tools are brittle rule engines. They work great for a specific document template, but the moment a vendor changes their invoice format or adds an extra field, the entire pipeline breaks. You end up with fragile regex patterns, hard-coded field positions, and expensive human intervention to patch things.

We wondered: What if instead of encoding rigid rules, we gave the automation system actual reasoning capability? What if it could handle documents it had never seen before, explain why it made decisions, and know when to escalate to a human rather than confidently making mistakes?

That's where the inspiration for FlowPilot came from. We wanted to build a system that replaces brittle rule engines with intelligent agent decision-making -- powered by modern LLMs.

But we also realized that calling a massive reasoning model (like Nemotron Ultra) on every single routine invoice would be wasteful. So we implemented something smarter: a two-tier routing pattern. Fast, cheap models (Nemotron Nano) handle the 90% of cases that are straightforward, and we only escalate to expensive reasoning models when something actually looks suspicious or ambiguous.

WHAT WE LEARNED

Two-Tier Routing is Powerful for Cost Efficiency

The key insight was implementing intelligent routing. This pattern drastically reduces inference costs without sacrificing accuracy on edge cases. We learned that the real value isn't in having the biggest model -- it's in being smart about which problems need which model.

OpenAI SDK Compatibility is a Superpower

Nebius Token Factory exposes an OpenAI-compatible API, which meant we could drop in NVIDIA Nemotron models without rewriting any client code. Just change the base_url and model name. This taught us how important API standardization is for rapid prototyping in the LLM space.

Mock Mode is Better Than Credentials Friction

Building a deterministic mock fallback for when NEBIUS_API_KEY isn't set was crucial. It meant the entire pipeline stays runnable without credentials, which is essential for demos, CI/CD testing, and onboarding. We'd do this again.

Agent "Hands" Need to be Concrete

We learned that agents without real side effects are just chatbots. By giving our agent actual tools (update_ledger, flag_for_review), the system becomes tangible and inspectable. Every action is logged, timestamped, and visible in the dashboard.

HOW WE BUILT IT

Architecture: Three Components

Backend (FastAPI + Python)
- main.py: FastAPI app exposing /process (runs full pipeline) and /ledger (view actions)
- nemotron_client.py: Thin wrapper around Nebius Token Factory API, handles Nano→Ultra routing
- tools.py: Agent's "hands" -- concrete actions like update_ledger() and flag_for_review()
NemotronClient: The Decision Engine
- Extract data from document using Nano (fast triage)
- Check if escalation is needed
- If needs_escalation or confidence < 0.75, call Ultra (deeper reasoning)
- Otherwise proceed with Nano output and approve
Frontend: Zero-Build Dashboard
- index.html: Self-contained dashboard (no build step)
- Real-time queue visualization
- Live reasoning trace showing which model is "thinking"
- Ledger view of all actions taken
- Gracefully falls back when backend is unavailable

Data Flow

Document → Nemotron Nano Classification & Extraction Check: Needs Escalation? YES → Nemotron Ultra Deep Reasoning NO → Proceed with Nano output Agent Takes Action (update ledger OR flag for review) Return Reasoning Trace + Follow-up Note

The Demo Loop

User clicks "Run Queue" on the dashboard
Frontend fetches sample documents (invoice_1.txt, invoice_2_anomaly.txt, receipt_c2201.txt)
For each document, POST /process is called
Backend routes to Nano or Ultra based on confidence + anomaly flags
Actions are written to ledger.json
Live trace is streamed back to frontend in real-time
Dashboard shows which model handled each case and why

CHALLENGES WE FACED

Challenge 1: Import Path Hell (Windows PowerShell Gotcha)

Problem: When we ran uvicorn main:app --reload --port 8000 from the project root, Uvicorn couldn't find the nemotron_client module because it was looking in the wrong directory.

Solution: We switched to relative imports (from .nemotron_client import NemotronClient) and called Uvicorn as python -m uvicorn backend.main:app --port 8000 from the project root. This ensured Python's module resolution worked correctly.

Lesson: Path management in multi-file Python projects requires care, especially when mixing relative and absolute imports. Relative imports + proper working directory = sanity.

Challenge 2: Mock Mode Completeness

Problem: How do we demo the system without real API credentials? Building a fake response that's realistic enough is non-trivial.

Solution: We built a deterministic mock that detects anomalies (high amounts, unusual payment terms), returns JSON that matches the real model's schema, and respects the needs_escalation flag logic. This meant the entire flow works without touching the real API.

Challenge 3: Reasoning Model Response Parsing

Problem: LLM outputs aren't always perfectly formatted JSON. Sometimes there are markdown fences, trailing newlines, or hallucinated keys.

Solution: We added robust JSON extraction that strips markdown code fences, attempts to parse the JSON, and falls back to a raw string if parsing fails.

Challenge 4: Ledger Concurrency

Problem: If multiple requests process documents simultaneously, writing to ledger.json could cause race conditions.

Solution: For a hackathon MVP, we kept it simple (single-threaded JSON writes work for the demo scale). In production, we'd use a database with transactional writes, file locking, or queue-based ledger updates.

Challenge 5: Frontend-Backend Communication During Demo

Problem: The frontend needs to work in two modes: offline mode (pure client-side demo) and online mode (call the real backend API).

Solution: We added graceful fallback logic in the frontend so that when the backend is unavailable, it falls back to mock responses. This means the demo works instantly even without the backend, but lights up with real data when it is running.

WHAT WE'D DO DIFFERENTLY (NEXT STEPS)

Add a proper database instead of JSON files for production ledger scale
Implement streaming for the reasoning trace so we see model thinking in real-time
Add document classification confidence scores for audit trails
Build a feedback loop so the system learns which edge cases need escalation
Cache repeated extractions to reduce API calls on duplicate documents

KEY TAKEAWAYS

Smart routing beats raw model size for real-world efficiency
Concrete agent actions matter -- logs, timestamps, side effects make it real
Mock modes unblock development -- don't let credentials become a blocker
OpenAI compatibility is huge -- it made multi-model swapping trivial
RPA needs reasoning, not just rules -- but reasoning doesn't need to be expensive on every case

FlowPilot demonstrates that the future of RPA isn't brittle templates - it's intelligent agents that reason about what they don't understand, and know when to ask for help.