WebForge

If you need to call it and it has no api, Forge It.
Main page
Dashboard
Create a new API

Inspiration

Getting structured data from the web usually means finding an API, learning a new data format, or writing a scraper that breaks the next time a page changes. We wanted to make that first step feel much simpler: describe the data you need in plain English and get an API you can actually use.

What it does

WebForge turns a plain-English request for public web data into a refreshable JSON API. It proposes a schema and a search plan, lets you choose the fields that matter, then discovers and extracts data from relevant public pages. Each record includes its source so the result is easy to inspect instead of being a black box.

You can use automatic discovery or provide specific URLs, choose a bounded search depth, refresh a dataset later, and retrieve records, schemas, and run history through stable endpoints. For facts that live across multiple pages, WebForge can combine complementary sources into a single record.

How we built it

We built WebForge with Next.js and TypeScript for the dashboard and API layer, MongoDB Atlas for persistent jobs and records, and a separate worker for queued runs and scheduled refreshes. OpenAI transforms each user request into a proposed schema and focused search plan, while Firecrawl handles public-web discovery, scraping, screenshots, and structured extraction. We designed the pipeline around transparency and reliability. Records retain source URLs, field-level provenance, extraction timestamps, and detailed run histories. When text extraction fails, the worker can use a bounded visual fallback to read the rendered page while applying the same schema validation, relevance, identity, and quality checks.

Challenges we ran into

The hard part was making web extraction useful without pretending it is always perfect. Pages can be sparse, off-topic, blocked, or inconsistent, and a broad search can easily surface something that looks relevant but is not. We added source review, schema confirmation before extraction, evidence checks for numeric prices, and clear partial-result states so missing data is visible rather than silently accepted.

We also had to balance flexibility with cost and reliability. Each run has limits on searches, scrapes, recovery attempts, vision fallbacks, and runtime, which keeps the system predictable while still giving it room to recover from bad sources.

Accomplishments that we're proud of

We’re incredibly proud that WebForge feels like a complete product rather than just a 36-hour hackathon demo. From a single plain-English request, WebForge can understand the data a user wants, propose a typed schema, let them approve or customize the fields, and transform messy websites into a live, structured API.

Users can provide exact URLs or let WebForge automatically discover relevant sources. They can choose between Focused, Balanced, and Deep search modes to control coverage, record volume, and Firecrawl credit usage. WebForge can combine information from multiple pages, remove duplicates, validate record identity, reject unrelated or suspicious results, and return honest partial data when some fields cannot be verified.

When conventional HTML extraction fails because a page is dynamic, obfuscated, or protected by anti-bot techniques, WebForge can fall back to visual extraction and interpret the rendered page. Every record retains its original sources, including field-level provenance, so users can see where individual values came from instead of blindly trusting an AI-generated response.

Users can watch each run progress from discovery through extraction, inspect errors and source decisions, and access the finished data through stable endpoints for records, schemas, and run history. APIs can be refreshed manually or kept current automatically with configurable schedules that can be paused and resumed. Failed refreshes preserve the previously verified data rather than replacing it with incomplete results.

We also built a full management dashboard, cost-aware extraction limits, quality checks, cancellation and recovery handling, a downloadable CLI for terminal and VS Code workflows, and a polished product website explaining the entire platform. WebForge does not simply scrape a page once—it creates a source-backed, refreshable API that developers can immediately use in real applications.

What we learned

We learned that trustworthy AI-powered data products need to show their work. Giving users control over the schema before extraction—and attaching sources to every resulting record—makes the system far more transparent and understandable. We also learned that when the web does not cooperate, honest partial results are much more valuable than overconfident or fabricated answers.

Building WebForge also taught us how difficult reliable web extraction can be. Websites change, relevant data is often spread across multiple sources, and traditional HTML parsing does not always work. By combining structured extraction, visual fallback, validation, provenance, and controlled search budgets, we learned how to build a system that can fail gracefully while still producing useful, verifiable data.

What's next for WebForge

Like you saw, WebForge is capable of so, so much—but, admittedly, like any 36-hour hackathon project, there is still so much more to do!

Our vision is for WebForge to become the API layer for everything on the web that doesn’t already have one. Next, we want to build self-healing extraction that adapts when websites change, smarter refresh schedules that reduce costs, confidence scores backed by multiple sources, and alerts when important data changes. We also hope to support authenticated websites, team workspaces, webhooks, database integrations, generated SDKs, and a marketplace of reusable API templates.

Ultimately, we want developers to stop building and repairing one-off scrapers. They should be able to describe the data they need and trust WebForge to discover it, verify it, keep it current, and deliver it wherever their applications need it.