OmniProcure

OmniProcure cuts 6-hour manual procurement to minutes — Amazon Nova Pro orchestrates supplier sourcing, compliance, and PO generation with vision QA and human-in-the-loop approval.

Comment

Inspiration

Enterprise procurement is broken at the execution layer.

A procurement manager at a mid-size manufacturer spends 6+ hours per week doing this:

  • Logging into 3–5 different supplier portals manually
  • Cross-referencing product catalogs, pricing, and stock
  • Running compliance checks against company procurement policies
  • Filling out purchase order forms by hand
  • Waiting for approvals via email

No ERP solves this. SAP and Oracle handle records — they don't act. The last mile of procurement — the actual browsing, checking, and ordering — is still entirely manual.

We built OmniProcure to eliminate that last mile.

What it does

OmniProcure is a multi-agent AI platform that takes a natural language procurement request and autonomously:

  1. Searches supplier catalogs using a Nova Act-powered browser agent
  2. Runs compliance checks against procurement policies using Amazon Nova Pro
  3. Reviews visual evidence (screenshots, product images) using Nova Lite's multimodal vision
  4. Generates a structured Purchase Order with a full audit trail
  5. Routes to a human approver via a real-time HITL dashboard before any order is placed

The agent pipeline completes each order in \(\approx 45\) seconds of autonomous execution. The procurement manager's active time drops from \(7.5\) min/order to \(0.75\) min of HITL review — a 10x reduction in active work per order.

$$ \text{Active Time Saved} = 50 \times (7.5 - 0.75) \text{ min/order} = 337.5 \text{ min/week} \approx 5.6 \text{ hrs/week} $$

A 50-order/week workload shrinks from ~6.25 hours of active manual work to ~37 minutes of agent-supervised review.

How we built it

Tech Stack

Layer Technology
Agent Orchestration Amazon Bedrock Strands SDK
LLM Amazon Nova Pro (via Amazon Bedrock)
Browser Automation Amazon Nova Act (headless Chromium)
Visual QA Amazon Nova Lite (multimodal vision)
Backend API FastAPI + Python 3.11
Auth Amazon Cognito (JWT, user pools)
Frontend Next.js + Tailwind CSS
Database SQLite (MCP-inspired data access layer)
Observability Amazon CloudWatch structured logs
Secrets AWS Secrets Manager

Agent Architecture

OmniProcure runs 5 specialized agents orchestrated by a central Strands controller:

┌─────────────────────────────────────────────┐
│         Strands Orchestrator                │
│         (Amazon Nova Pro)                   │
│  Decomposes request → delegates → retries   │
└──────┬──────────┬──────────┬────────────────┘
       │          │          │
  ┌────▼───┐ ┌────▼────┐ ┌───▼──────────┐
  │Catalog │ │Compli-  │ │  Actuator    │
  │ Agent  │ │ance     │ │  Agent       │
  │(ERP    │ │ Agent   │ │  (Nova Act)  │
  │ tools) │ │         │ │              │
  └────┬───┘ └────┬────┘ └───┬──────────┘
       │          │          │ screenshot
       └──────────▼──────────▼
              ┌───────────────┐
              │  Evidence     │
              │  Agent        │
              │ (Nova Lite    │
              │  Vision)      │
              └───────┬───────┘
                      │ verdict
               ┌──────▼──────┐
               │  HITL Gate  │
               │  Dashboard  │
               └─────────────┘

Orchestrator Agent (Nova Pro) — Decomposes the natural language request, delegates to worker agents, handles retries on capacity errors, and aggregates results into a final PO decision.

Catalog Agent — Queries the MCP-inspired ERP tool layer for SKUs, pricing, and supplier availability via typed tool calls. Returns structured product matches ranked by compliance fit.

Compliance Agent — Validates the order against retrieved procurement policies. Outputs APPROVED, FLAGGED, or REJECTED with reasoning and a confidence score.

Actuator Agent (Nova Act) — Drives a headless Chromium browser using intent-level instructions rather than CSS selectors. Captures screenshot evidence at each key step.

Evidence Agent (Nova Lite Vision) — Reviews Actuator screenshots multimodally. Confirms product name, quantity, price, and visual appearance match the original request before the PO is generated.

How We Used Amazon Nova

Nova runs the entire pipeline — not just one step:

  • Nova Pro — Central orchestrator reasoning across multi-step procurement decisions, producing structured JSON outputs for downstream agents
  • Nova Act — Intent-level browser automation; the instruction "Search for industrial gloves size Large and extract price and stock" works even when portal UI changes, because Nova Act reasons about intent rather than DOM structure
  • Nova Lite (multimodal) — Visual QA on checkout screenshots before PO generation, closing the trust gap between automation and human confidence

Every inference call is logged to Amazon CloudWatch with a structured trace per job_id, creating a fully replayable audit trail for compliance purposes.

MCP-Inspired Data Access Layer

Rather than injecting raw database schemas into agent prompts, OmniProcure uses typed Python tool functions that agents call directly — an MCP-inspired pattern ready to wrap into a real JSON-RPC MCP server in Phase 2:

@tool
def get_erp_inventory(item_name: str) -> dict:
    """Returns current inventory level for an item from the ERP cache."""
    ...

@tool
def get_procurement_policy(category: str) -> dict:
    """Returns active procurement policy rules for a spend category."""
    ...

This keeps prompts clean, makes tool contracts independently testable, and maps directly to a real MCP server in Phase 2.

Human-in-the-Loop Gateway

Every PO is blocked at a HITL gate before execution. The approver dashboard surfaces:

  • Full Strands agent reasoning trace (thinking tokens, tool calls)
  • Compliance verdict with confidence score and policy citations
  • Nova Act screenshots as timestamped visual evidence
  • Editable PO fields with one-click approve or reject

Only after explicit human approval does the system write the final purchase order.

Challenges we ran into

1. Live portals are brittle by design Real supplier portals have CAPTCHAs, session timeouts, and layout changes. We used a controlled demo portal for the hackathon, with Nova Act's intent-level instructions ready to generalize without selector rewrites. CAPTCHA detection returns a structured captcha_detected: true and routes to the HITL dashboard instead of crashing.

2. Context window management across 5 agents Five agents sharing a Strands orchestrator risks token bloat. We solved this with compact JSON tool contracts and _truncate safeguards — each agent receives only the fields it needs, keeping total prompt size bounded.

3. Nova Act sessions failing silently Headless browser sessions terminated mid-job without raising explicit exceptions. We built a session lifecycle manager that checkpoints job state so a failed session can resume from its last confirmed step rather than restarting the full pipeline.

4. Grounding compliance checks Early Nova Pro compliance runs hallucinated policy rules. We fixed this by injecting policy data exclusively through the get_procurement_policy tool — no policy text in the system prompt — forcing the agent to retrieve, not recall.

5. Streaming agent thoughts to a live UI Strands produces streaming tokens. Bridging those to a Next.js dashboard via Server-Sent Events required a careful async generator pipeline in FastAPI to prevent dropped frames during long agent runs.

Accomplishments that we're proud of

  • A fully working 5-agent orchestration pipeline using Amazon Bedrock Strands SDK that takes a plain English request and produces an auditable Purchase Order — end to end, without human intervention until the approval gate.
  • Nova Act integration that performs intent-level browser automation with screenshot capture, functioning even when portal UI changes — no hardcoded CSS selectors anywhere in the codebase.
  • A multimodal Evidence Review step using Nova Lite Vision that visually verifies procurement screenshots before any PO is generated — a level of trust verification absent from existing procurement automation tools.
  • A production-grade HITL approval dashboard that surfaces agent reasoning traces, compliance verdicts, and visual evidence to a human approver in real time via Server-Sent Events.
  • A clean MCP-inspired tool architecture that keeps every agent prompt free of raw schema injection and makes each tool contract independently testable.
  • End-to-end CloudWatch observability with structured traces per \(\text{job_id}\), giving enterprises a fully replayable audit trail from request to PO.

What we learned

  • Nova Act is genuinely different. Most browser automation breaks when a site updates its UI. Nova Act's intent-level instructions survived portal UI changes in our tests without any selector updates — this is real self-healing automation.
  • Strands makes multi-agent orchestration readable. The agent loop, tool calling, and streaming are handled cleanly by the SDK — we focused on procurement logic rather than infrastructure plumbing.
  • HITL is not a limitation — it is the product. We initially treated human approval as a workaround for AI errors. Enterprises require this audit gate for compliance, liability, and CFO sign-off. It is the differentiator, not the weakness.
  • Vision QA closes the trust gap. Adding screenshot-based evidence review via Nova Lite's multimodal capability was the single change that made the system feel enterprise-ready rather than demo-grade.
  • Retrieval beats recall for compliance. Forcing the Compliance Agent to retrieve policy rules via tool calls — rather than relying on prompt-injected context — eliminated hallucinated policy verdicts entirely.

What's next for OmniProcure

Phase 2:

  • Portal Registry — Config-driven library supporting any supplier portal, with per-portal login macros and OTP/MFA HITL routing
  • PostgreSQL on Amazon RDS — Replace the edge SQLite cache with multi-tenant RDS and per-tenant schemas; the existing data access interfaces require zero changes to agent code
  • Real MCP Server — Wrap the ERP tool layer in a proper JSON-RPC MCP server, making OmniProcure a standards-compliant MCP host

Phase 3 (Production SaaS):

  • Invoice Reconciliation Agent and Contract Risk Agent built on the same Strands platform
  • Billing and metering — Per-tenant Nova Act minute tracking with Stripe integration
  • SOC 2 compliance from day one, leveraging the existing CloudWatch audit trails already embedded in the pipeline

Target market: Mid-market manufacturers and distributors running \(50+\) manual orders per week — the segment where no ERP has an API for the portals they actually use every day.

OmniProcure is built for the 80% of enterprise procurement that happens in portals no ERP has ever automated — and Phase 2 is already designed.

Built With

Share this project:

Updates