Flow Pilot

Describe any workflow in plain English. FlowPilot plans it, executes it, and fixes itself — powered by Amazon Nova.

Comment

Inspiration

Every business has repetitive web-based workflows: checking competitor prices, monitoring brand mentions, researching leads, processing invoices. Tools like Zapier or n8n solve some of this, but they require upfront technical configuration — connectors, triggers, action blocks — and they still can't navigate arbitrary websites like a human would.

The question we asked: What if you could just describe what you want done, and an AI agent figures out the rest?

That's FlowPilot. You type "Check Sony headphone prices on Amazon and eBay every morning and flag if the difference is over 15%." FlowPilot turns that sentence into a running, scheduled, browser-powered workflow — no configuration wizard, no drag-and-drop connectors required.

What It Does

FlowPilot is an AI-powered workflow automation platform with three layers of intelligence — all backed by Amazon Nova on AWS Bedrock:

1. Natural Language Planning (Nova 2 Lite)

Type a description of your workflow in plain English. Nova 2 Lite reasons over your description and decomposes it into a structured sequence of browser automation steps: navigate, click, type, extract, wait, and conditional. It generates realistic targets, URLs, and logic branches — no manual step setup needed.

2. Live Browser Execution + Multimodal Analysis (Playwright + Nova Pro)

Steps execute in a real headless Chromium browser via Playwright. For extract steps, Nova Pro analyzes the live page — reading its content and structuring it into clean JSON (products with prices and ratings, news articles with authors, LinkedIn profiles with company details). Screenshots are captured at each step and streamed to the dashboard in real time via Server-Sent Events (SSE).

3. AI Self-Healing (Nova 2 Lite)

When a step fails (element not found, page layout changed, CAPTCHA encountered), FlowPilot doesn't immediately ask the user for help. First, it invokes Nova 2 Lite with the error message and step context to generate a fix — a corrected selector, an alternative target, or a retry strategy. If the self-healed step succeeds, execution continues automatically. Only if self-healing also fails does the system surface the error to the user with options to retry, skip, or abort.

4. Conditional AI Reasoning (Nova 2 Lite)

Workflows support conditional branching steps. Nova 2 Lite evaluates the condition against the previous step's extracted data (e.g., "Is amazon_price less than ebay_price?") and decides which branch to take — turning static scripts into adaptive agents that react to what they actually find.

Additional Features

  • Visual Step Editor — drag-and-drop step reordering with @dnd-kit
  • Cron Scheduling — automated workflow runs via APScheduler
  • 5 Pre-built Templates — Invoice Processing, Lead Research, Social Media Monitor, Competitor Price Check, Daily News Digest
  • Real-time Dashboard — live run progress, step-by-step results, success rate charts
  • Error Recovery UI — interactive modal to retry, skip, or abort failed steps

How We Built It

Architecture

User (Dashboard) ──► FastAPI Backend ──► Amazon Bedrock
       │                    │                   │
       │  SSE Events        │  SQLAlchemy        │  Nova 2 Lite (Planning + Self-Healing)
       │  (live updates)    │  (SQLite/Postgres)  │  Nova Pro (Multimodal Extraction)
       │                    │                   │  Nova Act SDK (Browser Automation)
       ▼                    ▼                   ▼
  React + Vite        Workflow Engine      AI Services
  TanStack Query      APScheduler         Step Execution
  Recharts            SSE Streaming       Screenshot Analysis
  @dnd-kit            Error Recovery      Conditional Logic

Nova Model Usage

Model Purpose Why This Model
Amazon Nova 2 Lite NL → JSON step planning Fast text reasoning, low latency, structured JSON generation
Amazon Nova Pro Page content → structured data Multimodal understanding of live page screenshots and HTML content
Amazon Nova 2 Lite Self-healing failed steps Reasoning over error context to generate corrective actions
Amazon Nova 2 Lite Conditional branch evaluation Evaluating extracted data against workflow logic conditions

Backend

  • FastAPI with async SQLAlchemy for non-blocking DB operations
  • APScheduler for cron-based workflow scheduling
  • SSE-Starlette for real-time Server-Sent Events streaming to the frontend
  • Playwright for headless Chromium browser automation, with CAPTCHA fallback (auto-redirects to DuckDuckGo when blocked)
  • AWS Boto3 for Bedrock API calls, with throttle detection and retry logic

Frontend

  • React 18 + TypeScript + Vite
  • TanStack Query for server state management
  • Recharts for run history and success rate charts
  • @dnd-kit for drag-and-drop step reordering
  • Custom useSSE hook for live EventSource connection to the backend

Graceful Degradation

The execution engine runs in three priority tiers:

  1. Real browser + Nova AI — full live automation with AI-powered extraction
  2. Nova AI only — generates realistic structured data when browser isn't available
  3. Simulation fallback — deterministic fallback so the full UI experience works even without AWS credentials

Challenges We Ran Into

Bot detection and CAPTCHAs — Modern sites aggressively block headless browsers. We built a detection layer that identifies bot-block pages by checking for known CAPTCHA keywords and response patterns, then automatically falls back to DuckDuckGo for searches rather than failing the step.

Structuring unstructured web content — Raw DOM content from Playwright is noisy. We built a content structuring pipeline that uses URL patterns and extraction descriptions to route raw content through specialized parsers (product pages, search results, news, LinkedIn profiles) before passing it to Nova Pro — reducing the token burden on the model and improving JSON output quality.

SSE with asyncio and concurrent runs — Managing multiple simultaneous workflow runs, each with their own SSE fan-out queue, required careful design. We use a class-level dict of asyncio.Queue lists in ExecutorService, where each subscriber gets its own queue, and events are broadcast to all connected clients for a given run.

Self-healing loop safety — A naive self-healing implementation can loop infinitely. We designed self-healing as a single best-effort attempt: Nova suggests a fix, we apply it once, and if it fails again, we escalate to the user rather than retrying.

Accomplishments We're Proud Of

  • End-to-end agentic loop — A user types one sentence and gets back a running, scheduled workflow with live step-by-step results and AI-powered error recovery. The entire pipeline from NL description to structured extracted data is powered by Nova.

  • AI self-healing in production — Using Nova to fix its own execution failures mid-run is a genuinely novel recovery mechanism that makes workflows dramatically more robust without user intervention.

  • Multimodal extraction pipeline — Nova Pro reading live Playwright screenshots to extract structured data (prices, profiles, articles) bridges the gap between browser automation and intelligent data extraction in a way that neither tool achieves alone.

  • Graceful 3-tier fallback — The system degrades gracefully from full browser+AI → AI-only → simulation, meaning the full dashboard experience is always demonstrable regardless of environment.

What We Learned

  • Amazon Nova 2 Lite is exceptionally strong at structured JSON generation with low latency — ideal for the planning loop where speed matters more than depth.
  • Nova Pro's multimodal capabilities shine when given both page screenshots and extracted text — combining visual and textual signals significantly improves extraction accuracy.
  • The hardest part of agentic browser automation isn't the AI — it's the web's adversarial environment (CAPTCHAs, dynamic layouts, bot detection). Robust fallback strategies are as important as the AI layer.

What's Next

  • Nova Act SDK deep integration for more precise element targeting
  • Workflow chaining — output of one workflow feeds input to another
  • Persistent variable store — workflows that accumulate data across runs (price history, sentiment trends)
  • Email/Slack notifications when conditional thresholds are exceeded
  • Team collaboration — shared workflows and run history across users

Screenshots

Dashboard — Live metrics, run history chart, success rate breakdown Dashboard

Workflows List — Quick Create with AI banner, workflow cards with step previews Workflows

AI Step Generator — Natural language input, example prompts, Generate Steps button NL Input

Workflow Builder — Run stats, cron schedule, webhook trigger, visual flow diagram Workflow Builder

Templates Marketplace — 5 pre-built templates across Finance, Sales, Marketing, Research Templates

Run Viewer — Live progress bar, AI Summary, step-by-step list with status icons Run Viewer

Extract Step Result — Nova Pro structured JSON from live Amazon page (products, prices, ratings) Extract Result

Conditional Step Result — Human-readable branch outcome with Nova AI evaluation Conditional

Built With

Share this project:

Updates