Webis

Webis is a modular, plugin-based AI data pipeline that supports data acquisition, cleaning, structured extraction, and RAG generation, solving the upstream supply problem for AI applications.

Comment

Inspiration

The inspiration for Webis came from a critical observation during our research into enterprise AI adoption: organizations are investing heavily in large language models, yet neglecting the information supply chain that powers them.

We discovered a systemic "upstream crisis" plaguing AI AI applications:

  • Fragmented Data Sources: News crawlers, academic paper retrieval, code search, and internal document parsing operate as isolated scripts—no shared standards, no reusability
  • Format Chaos: HTML noise, PDF OCR errors, encoding issues, and inconsistent schemas create a cleaning nightmare
  • Quality Blindness: Without traceability or evidence chains, AI systems generate "confidently wrong" answers that are impossible to audit
  • Delivery Inconsistency: The same pipeline must be rebuilt separately for CLI tools, web services, and UI applications
  • Data Value Erosion: Processed data is typically discarded after immediate use, creating a recurring cost of re-acquisition, re-cleaning, and re-validation. Organizations fail to recognize that cleaned, structured, and traceable data is a compounding asset—its value increases with volume, linkage, and historical depth

A revealing case study: A financial institution's research assistant cited a two-year-old news article when answering a query about "latest earnings risks"—causing real investment losses. The system had no timestamp validation, no source tracking, and no quality gates. Worse, the cleaned data from previous runs wasn't retained, forcing the system to re-crawl potentially outdated sources rather than leveraging a curated knowledge base with temporal awareness.

The core insight: AI applications don't fail because models are dumb—they fail because the data feeding them is untraceable, ungoverned, unfit for production, and treated as disposable rather than a persistent strategic asset. Webis treats data as infrastructure: every processed document becomes part of an evolving knowledge foundation that appreciates in value through reuse, cross-referencing, and temporal accumulation.

Webis was born from a simple mission: transform AI data pipelines from "fragile script chains" into modular, observable, production-grade infrastructure that preserves and amplifies data value over time.

What it does

Webis is a plugin-based pipeline framework that standardizes the journey from raw data to AI-ready knowledge. It treats "data acquisition → content cleaning → structured extraction → RAG knowledge base" as a unified, configurable production line.

Core Capabilities

1. Unified Data Model (WebisDocument) Every piece of data carries:

  • Content: Cleaned, denoised text
  • Metadata: Source URLs, timestamps, authors, data types
  • Processing Trace: Complete audit trail of which plugins touched the data, when, and with what quality scores

2. Four-Plugin Architecture

Plugin Type Responsibility Examples
Sources Data acquisition Tavily search, GitHub API, web crawlers, file readers
Processors Content normalization HTML cleaning, deduplication, chunking, embedding, anonymization
Extractors Structured generation LLM-based JSON extraction with schema validation
Outputs Knowledge delivery RAG vector stores, structured reports, API responses

3. Intelligent Orchestration (CrawlerAgent)

  • Dynamically selects optimal data sources based on task intent
  • Balances load across tools with automatic quota adjustment
  • Degrades gracefully from LLM planning to static tool chains when needed

4. Dual-Output Production Same cleaned data stream simultaneously produces:

  • Structured JSON: For business systems, automation, and audit trails
  • RAG Knowledge Base: For LLM context retrieval with full provenance

5. Multi-Modal Delivery One pipeline, three consumption modes:

  • CLI: For developers and debugging
  • Streamlit UI: For collaborative exploration

How we built it

Phase 1: Foundation (Unified Data Model)

We started by rejecting the "script collection" approach. Instead of chaining independent crawlers, we enforced a mandatory data contract: every component must output a WebisDocument with content, metadata, and trace fields.

Gemini as Architecture Partner: From day one, Google Gemini served as our collaborative architect. We used Gemini to:

  • Design the core schema: Iteratively refining the WebisDocument structure through natural language debates about what metadata fields were essential for traceability
  • Generate base classes: Prompting Gemini with "Design a Python abstract base class for a document processing pipeline that enforces immutability and audit trails" and refining its outputs
  • Validate design decisions: Using Gemini to simulate edge cases and identify missing fields in our data model before implementation

This constraint felt heavy initially but proved transformative—it enabled plugin interchangeability, pipeline observability, and downstream trust.

Phase 2: Plugin Architecture (Boundary Design)

We deliberately limited plugin types to four, resisting the temptation to create "kitchen sink" extensibility. Each plugin type has a strict interface:

class SourcePlugin(BasePlugin):
    @abstractmethod
    def fetch(self, query: str, limit: int = 10) -> Iterator[WebisDocument]:
        """Must return standardized document stream"""

Gemini as Code Generator: This phase was heavily accelerated by Gemini's code generation capabilities:

  • Interface scaffolding: We provided Gemini with high-level descriptions like "Create a processor plugin interface that accepts a WebisDocument and returns a modified WebisDocument with processing metadata" and received production-ready abstract classes with type hints and docstrings
  • Template generation: Gemini generated complete plugin templates for all four types, including boilerplate for error handling, logging, and metadata preservation
  • Cross-language consistency: When we needed similar interfaces in TypeScript for the UI layer, Gemini translated the Python patterns while maintaining semantic equivalence
  • Refactoring automation: We fed Gemini chunks of prototype code and asked it to "extract common functionality into shared mixins," receiving well-structured inheritance hierarchies

Key design decision: Plugins are stateless and composable. A source doesn't know about processors; an extractor doesn't know about outputs. This enables true mix-and-match flexibility.

Phase 3: Agent Intelligence (Adaptive Orchestration)

Static pipelines failed for "exploratory" tasks—researching unknown domains where you don't know which sources to use. We built CrawlerAgent with:

  • LLM-based planning: Generates execution plans in JSON format
  • Tool registry introspection: Automatically discovers available plugins
  • Dynamic resource allocation: Redistributes quotas when sources underperform
  • Fault tolerance: Individual plugin failures don't crash the pipeline

Gemini as Logic Designer: The CrawlerAgent core was co-developed with Gemini:

  • Prompt engineering: We used Gemini to iterate on system prompts for agent planning, testing variations like "You are a data acquisition planner..." vs "Act as an intelligent router for information sources..." and measuring output quality
  • JSON schema generation: Gemini drafted Pydantic models for agent outputs, including complex nested structures for execution plans with fallback chains
  • Algorithm design: We described the quota rebalancing problem in natural language, and Gemini proposed weighted round-robin algorithms with performance decay factors
  • Error handling logic: Gemini generated comprehensive exception hierarchies and retry strategies with exponential backoff

Counter-intuitive choice: We avoided hardcoding tool priorities. This sacrifices predictability for genuine adaptability—new plugins are automatically utilized without code changes.

Phase 4: Dual-Track Output (RAG + Structuring)

The hardest architectural decision: should we optimize for structured data or RAG context? We chose both.

By splitting cleaned data into parallel processing tracks:

  • Track A: Schema-validated JSON for deterministic business logic
  • Track B: Semantic chunks with metadata for vector retrieval

This eliminates the common anti-pattern of "re-running extraction for RAG" or "maintaining separate pipelines for different outputs."

Gemini as Optimization Consultant: For the dual-track system, Gemini provided critical implementation support:

  • Chunking strategies: We prompted Gemini with academic papers on semantic chunking and asked for Python implementations respecting sentence boundaries and context windows
  • Parallel processing patterns: Gemini generated async/await patterns for simultaneous Track A and Track B processing, including shared memory management to avoid data duplication
  • Schema alignment: Gemini helped design JSON schemas that could be losslessly converted to vector embeddings and back, ensuring consistency between structured and semantic outputs
  • Performance benchmarking: Gemini wrote profiling scripts to measure token consumption differences between single-track and dual-track approaches, confirming our 40% efficiency claim

Challenges we ran into

Challenge 1: The HTML "Noise War"

Problem: Modern web pages are hostile to extraction—Shadow DOM, JavaScript rendering, anti-bot measures, infinite scroll.

Our solution: Tiered cleaning strategy

  • Layer 1: Heuristic filtering (BeautifulSoup removing script/style tags)
  • Layer 2: LLM semantic extraction for main content identification
  • Layer 3: Mojibake detection and Chinese encoding repair

Lesson: Data quality is non-negotiable. One "good enough" cleaning job poisons downstream RAG with hallucination-prone context.

Challenge 2: Agent Hallucinations

Problem: CrawlerAgent would claim to invoke non-existent tools or generate malformed JSON.

Our solution: Defense in depth

  • Pydantic schema validation for all LLM outputs
  • Pre-execution tool existence verification
  • Double degradation: regex extraction → static fallback chain

Lesson: Never trust LLM outputs blindly. Production reliability requires multiple safety nets.

Challenge 3: RAG Context Fragmentation

Problem: Naive chunking (fixed character counts) destroyed semantics—premises and conclusions separated, arguments split mid-sentence.

Our solution: Semantic-aware chunking

  • Respect sentence and paragraph boundaries
  • Embed full source metadata in every chunk
  • Overlapping windows between adjacent chunks

Lesson: RAG quality ceiling is determined by chunking strategy, not embedding model sophistication.

Challenge 4: Plugin Versioning Hell

Problem: As plugins multiplied, dependency conflicts and breaking changes emerged.

Our solution: Strict interface contracts + semantic versioning

  • Plugins declare compatibility ranges
  • Pipeline validates plugin signatures at load time
  • Configuration schemas enforce valid parameter combinations

Accomplishments that we're proud of

1. True Plugin Interchangeability

We can swap from Tavily to Bocha to Exa search without touching pipeline code—just change a configuration string. This proves the architecture's abstraction strength.

2. Production-Grade Observability

Every output includes complete provenance: which sources were queried, which cleaning steps were applied, which LLM prompts were used, and quality scores at each stage. Debugging data pipelines no longer requires archaeology.

3. Dual-Output Efficiency

Parallel production of structured JSON and RAG knowledge bases eliminates redundant processing. Benchmarks show 40% reduction in token consumption compared to running separate extraction pipelines.

What we learned

Lesson 1: Constraints Enable Scale

The strict WebisDocument schema felt bureaucratic early on. In retrospect, it was the single most important decision—enabling plugin ecosystems, cross-team collaboration, and production governance.

Lesson 2: Flexibility Requires Boundaries

Counter-intuitively, limiting plugin types to four (rather than allowing arbitrary extensions) increased system flexibility. Clear boundaries prevent "spaghetti architecture" and enable genuine composition.

Lesson 3: Human-in-the-Loop is Non-Negotiable

Fully automated pipelines fail edge cases. We built explicit feedback hooks: when quality scores drop below thresholds, the system pauses for human review rather than producing garbage output.

What's next for Webis

Phase 1: Foundation (Current)

  • ✅ Unified data models and processing traces
  • ✅ Plugin-based architecture with four core types
  • ✅ Multi-modal delivery (CLI/UI/API)

In Progress: Enhanced plugin marketplace with version management, dependency resolution, and automated testing.

Phase 2: Knowledge Lake (Next 6 months)

Goal: Data should never "disappear after processing"

  • Persistent storage for structured results (metadata/business databases)
  • Vector database integration for RAG assets
  • Knowledge graph construction for entity relationship tracking
  • Cross-pipeline data reuse and discovery

Phase 3: Application Factory (12 months)

Goal: Evolve from "pipeline tool" to "capability platform"

  • Pre-built templates for common use cases (research assistants, intelligence radars, monitoring dashboards)
  • Visual pipeline builder for non-technical users
  • Automated report generation with source citations
  • Integration marketplace for downstream systems (Slack, Notion, CRMs)

Built With

Share this project:

Updates