Executive Report
Home page
Data Warehouse
Workflow Adapter
Live Audit Trail
Live Schema Validation

Orbit — Autonomous Goal-Driven Web Data Operations

"Set the goal. Walk away."
Orbit transforms natural-language objectives into recurring, verifiable web-data pipelines with autonomous self-correction, multi-sink delivery, and complete provenance audit trails.

Inspiration

In 2022, while building a platform to help students discover opportunities, I ran into a major roadblock: I needed to aggregate scholarships, fellowships, and academic grants across hundreds of university and government websites. As a software engineer without a specialized data science or scraping background, I had to take dedicated web scraping courses just to get started.

Over the following weeks, development turned into an endless operational grind. Every time a target website changed its DOM structure, CSS classes, or pagination layout, the scrapers silently broke. When scrapers stalled mid-execution or hit anti-bot CAPTCHAs, entire pipelines collapsed. Even after extracting raw HTML, I had to manually write regex rules, parse unstructured tables, configure database sinks, build email alerts, and constantly monitor page health. I was spending 80% of my time babysitting brittle scrapers and only 20% building product value.

This is a universal pain point across data science, engineering, market intelligence, and operations teams worldwide. In the modern era of autonomous AI, developers and data teams should not have to write or maintain brittle CSS/XPath selectors, manage proxy rotations, or manually wire extraction plumbing.

We built Orbit to solve this: an autonomous agentic platform where you state your high-level objective in plain English, and the agent discovers the sources, navigates layout shifts, extracts structured records, validates anomalies, and delivers verified data to PostgreSQL, S3, Slack, and executive PDF dossiers.

What It Does

Orbit replaces traditional, brittle web scrapers with an end-to-end autonomous agent loop:

Natural Language Goal Planning: Operators provide a high-level goal (e.g., "Track enterprise AI funding announcements over $10M, validate funding amounts, and export to PostgreSQL and S3 daily"). Orbit automatically induces structured Pydantic schemas, determines search queries, and constructs an executable DAG.
Autonomous Source Discovery & Resilient Retrieval: Dynamically finds target URLs via search engine discovery and fetches multi-format pages (HTML, PDF, DOCX) through intelligent proxy rotation and anti-bot mitigation.
Zero-Selector LLM Extraction: Uses Gemini / LLM multimodal reasoning to parse complex visual layouts, extract nested tabular data, and normalize attributes into strictly typed JSON records—without a single XPath or CSS selector.
Self-Correction & Statistical Anomaly Validation: The agent self-evaluates extracted records against field constraints and statistical distributions. If anomalies or missing attributes are detected, it autonomously triggers reasoning retry loops to correct itself.
Multi-Sink Delivery & Synthesis:
- Database Sinks: Direct upsert into PostgreSQL, MySQL, and customer data warehouses.
- Cloud Object Storage: Automatic archival of raw payloads and JSON lines to Amazon S3 / MinIO / Cloudflare R2.
- Executive PDF Dossiers: Compiles branded executive briefing reports with automated PII masking and verifiable data citations.
- Team Alerts: Sends Slack notifications and transactional email alerts with signed artifact download links.
Complete Provenance Audit Trail: Every single record retains source URLs, timestamp verification, layout snapshots, and agent reasoning logs for compliance and auditability.
Model Context Protocol (MCP) Server Integration: Built-in MCP server exposing Orbit's complete capabilities (tools, orbc:// resources, and prompts) to external AI assistants (Claude Desktop, Cursor, Antigravity, VS Code, Windsurf) over both standard stdio and remote SSE (Server-Sent Events) transports.
Omnichannel Ergonomics: Available via a Svelte 5 Web UI, a Go CLI binary (orbc), an MCP Server, and a headless FastAPI Daemon.

How We Built It

Orbit is engineered with a clean, decoupled architecture:

Reasoning & Agent Core (Python 3.12 & FastAPI):
- Powered by Google Gemini 3.7 (Flash) via Agents Strand layer for high-speed dynamic schema induction, structural extraction, and reasoning recovery.
- AsyncIO event bus and Server-Sent Events (SSE) stream real-time stage transitions, reasoning thoughts, and telemetry directly to clients.
- SQLAlchemy 2.0 and PostgreSQL / SQLite handle relational automation DAGs, execution runs, and provenance audit records.
Model Context Protocol (MCP) Server (orbit.server / orbc-mcp):
- Implements standard MCP tools (create_automation, run_automation, execute_goal, query_extracted_data, get_run_details, list_recurring_schedules).
- Provides live orbc:// dynamic resources (orbc://automations, orbc://runs/{run_id}) for contextual retrieval in LLM IDEs.
- Dual-mode transport support: local sub-process stdio mode and production HTTP/ASGI SSE mode (/sse, /messages/).
Operator CLI (orbc in Go):
- Pure static, cross-platform binary (CGO-free) built with Cobra and Viper.
- Implements strict configuration precedence (CLI Flags $\to$ Environment Variables $\to$ ~/.orbc/config.yaml $\to$ Compile-Time LDFLAGS).
- Real-time SSE telemetry streaming (orbc watch <id>), full workflow DAG deployments, and multi-format data exports (table, json, csv).
Frontend Web Application (Svelte 5):
- Built using Svelte 5 Runes ($state, $derived, $props, $effect) and Tailwind CSS.
- Real-time SSE connection for live run telemetry, interactive pipeline DAG designer, and dynamic telemetry charts.
Provider-Agnostic Adapter Matrix:
- Clean boundary separation between platform execution credentials and per-mission destination sinks (S3 storage, Slack webhooks, SMTP/transactional email, Document Converters).

Challenges We Ran Into

Non-Deterministic Web Layouts vs. Strict Schemas: Web pages are messy, unstructured, and continuously changing. We designed a two-phase verification engine: Phase 1 uses LLM extraction with structured schema bounds, and Phase 2 runs deterministic statistical anomaly validation. When validation fails, the agent inspects the error reasoning and retries extraction with alternative layout strategies.
Standardizing the Model Context Protocol (MCP) Lifecycle: Ensuring the MCP server seamlessly supports both local sub-process standard I/O (for desktop IDEs) and remote asynchronous SSE streaming with full error isolation and secret masking.
Real-Time Bidirectional Telemetry: Streaming agent reasoning steps, stage transitions, and extracted data rows simultaneously across multiple distinct interfaces (Svelte 5 Web UI, Go CLI terminal, and MCP streaming clients) required a robust, thread-safe SSE event broadcast architecture.
Strict Provider-Agnostic Architecture: To prevent vendor lock-in, we established architectural guardrails ensuring all components (LLM engines, search discovery, web unblockers, cloud storage, document converters) operate behind standard abstract interfaces with zero hardcoded provider names.

Accomplishments That We're Proud Of

True "Goal-In, Verified-Data-Out" Autonomy: Created an operational system that completely eliminates manual selector writing and maintenance.
First-Class MCP Integration: Bridged Orbit directly into the broader agent ecosystem, allowing any MCP-compliant AI assistant to formulate goals, trigger recurring pipelines, and query live data.
Comprehensive Multi-Platform Experience: Delivered a unified, polished experience across a Svelte 5 Web UI, a headless REST/SSE API, an MCP Server, and a pure Go CLI (orbc).
Enterprise-Grade Provenance: Every record carries verifiable provenance—source URL, extraction timestamp, confidence score, and agent decision trail.
Automated Executive Dossiers: Generates fully synthesized, PII-masked PDF intelligence briefings delivered straight to executive Slack channels and S3 buckets.

What We Learned

Agent Guardrails are Essential: Autonomous agents thrive best when balanced with deterministic guardrails. Combining LLM reasoning for extraction with strict Pydantic validation and statistical anomaly checks produces production-grade reliability.
Provenance Builds Trust: In automated data operations, speed is meaningless if the data cannot be audited. Providing complete reasoning trails and source citations gives operators confidence to "walk away."
Interoperability Multiplies Value: By implementing the Model Context Protocol (MCP), Orbit isn't just a standalone tool—it acts as an autonomous web data subsystem for any AI coding assistant or agent swarm.

What's Next for Orbit

Multi-Agent Collaborative Swarms: Specializing sub-agents into dedicated roles (Scout Agent for deep web navigation, Extractor Agent for multimodal parsing, and Auditor Agent for compliance checks).
Browser Action Sidecar: Enabling browser-use agents for complex multi-step forms, authenticated portal navigation, and dynamic Single Page Applications (SPAs).
Direct Vector & RAG Indexing Sinks: Adding native embedding and vector database sinks (Pinecone, Qdrant, Chroma) to feed extracted web data directly into enterprise generative AI knowledge bases.