Webscrapper

Just for searching things locally....

Comment

Federated Search Engine

A federated, peer-run web search network. Clone it, run one command, and get your own crawling + indexing + search node. Multiple nodes automatically divide the web into shards so they don't crawl the same sites.

Tech stack: Go (crawler + API) · Rust/Tantivy (search index) · Python (ML re-ranking) · PostgreSQL · Redis · HTMX (dashboard)

Quick Start

Requirements: Docker Desktop (with WSL2 on Windows) or Docker Engine on Linux/Mac.

git clone https://github.com/your-org/federated-search
cd federated-search
chmod +x start.sh
./start.sh

That's it. Open http://localhost:8080/dashboard in your browser.

  • Your API key is shown on the dashboard (and saved to data/keys/.api_key)
  • Start crawling from the Crawl tab, or via the API
  • Search immediately with whatever's been crawled

Usage

Dashboard

Open http://localhost:8080/dashboard to:

  • Monitor crawl progress (pages/min, frontier depth, storage used)
  • Start/stop/pause crawling
  • Manage API keys
  • Toggle local vs federated search mode
  • Test queries with live formatted results

API

# Search (returns links by default)
curl -X POST http://localhost:8080/v1/search \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "distributed systems", "format": "links", "limit": 10}'

# Get a page as Markdown
curl "http://localhost:8080/v1/page?url=https://example.com&format=markdown" \
  -H "Authorization: Bearer YOUR_KEY"

# Get a page as structured JSON
curl "http://localhost:8080/v1/page?url=https://example.com&format=structured_json" \
  -H "Authorization: Bearer YOUR_KEY"

Response formats: links · raw_html · markdown · structured_json

Crawl Control

# Seed from the built-in seed list
docker compose exec crawler ./crawler seed --file /seeds.txt

# Start crawling
docker compose exec crawler ./crawler crawl

# Or control from the dashboard

Stop / Pause

./pause.sh   # Stops crawler only — search keeps working
./stop.sh    # Full teardown (requires typing "DELETE EVERYTHING")

Configuration

Copy .env.example to .env and edit:

POSTGRES_PASSWORD=your-password
COORDINATOR_URL=          # Leave empty for single-node mode

All other settings (crawl speed, storage limits, search mode) are configurable from the dashboard or via PATCH /api/internal/settings.

Architecture

┌─────────────────────────────────────────┐
│              Your Machine               │
│                                         │
│  ┌──────┐  ┌──────┐  ┌──────────────┐  │
│  │  UI  │  │ API  │  │   Crawler    │  │
│  │ HTMX │◄─│  Go  │  │ Go + Colly  │  │
│  └──────┘  │  +   │  │ + chromedp  │  │
│            │ chi  │  └──────┬───────┘  │
│            └──┬───┘         │          │
│               │      ┌──────┴───────┐  │
│         ┌─────┴────┐ │   Postgres   │  │
│         │ Tantivy  │ │  (frontier,  │  │
│         │  Rust    │ │  pages,      │  │
│         │ indexer  │ │  link graph) │  │
│         └──────────┘ └──────────────┘  │
│                                         │
│              Redis (cache, dedup)       │
└───────────────────────┬─────────────────┘
                        │ federated queries
              ┌─────────▼──────────┐
              │    Coordinator     │
              │  (shard registry)  │
              └─────────┬──────────┘
                        │
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
       Node B        Node C        Node D

See docs/architecture.md for full detail.

Phases

Phase Status Description
0 — Foundations ✅ Schemas, Docker, CI, scripts
1 — Crawler 🔄 Colly fetcher, frontier, extraction, dedup
2 — Indexing ⏳ Tantivy BM25, Go API, multi-format responses
3 — Dashboard UI ⏳ HTMX control panel
4 — Coordinator ⏳ Shard assignment, peer registry
5 — Federation ⏳ Fan-out queries, circuit breaker
6 — ML Ranking ⏳ PageRank + semantic re-ranking
7 — Hardening ⏳ Packaging, security, observability

Documentation

License

MIT — see LICENSE

Built With

Share this project:

Updates