Website Category Classifier

Know what a website is about before you click — an AI API that classifies any URL into 11 content categories in real time.

Comment

Inspiration

Content filtering and brand safety are everywhere — parental control apps, ad networks, EdTech platforms — yet most of them rely on outdated blocklists or slow manual review. I wanted to see if a lightweight transformer model could classify any website by its actual content, not just its domain name, and package it as something developers could plug into their own products in minutes. What started as a personal curiosity about NLP quickly turned into a real question: could I build something people would actually pay to use?

What it does

WCC takes any URL and returns its category — Adult, Arts, Business, Education, Gaming, Health, Kids, Lifestyle, News, Recreation, or Technology — along with a confidence score. It scrapes live page content (not just the domain name), runs it through a fine-tuned DistilBERT model, and returns a JSON response in seconds. It's live on RapidAPI with Free, Basic, and Pro tiers, so anyone can integrate it into a parental control app, ad-verification pipeline, or content-moderation tool with a single API call.

How we built it

The system is split into two microservices for scalability: wcc-scraper, a Playwright-based service that fetches and cleans real page content (with a Browserless.io fallback for bot-protected sites), and wcc-classifier, which runs inference with a fine-tuned distilbert-base-uncased model hosted on Hugging Face. A FastAPI layer handles routing, auth, and fallback logic, sitting in front of both services. The model itself went through several iterations — starting as a TF-IDF + SVM baseline before being upgraded to DistilBERT for much stronger generalization. Hosting moved from Hugging Face Spaces, to Railway, and is now migrating to Google Cloud Run for permanent free-tier stability.

Challenges we ran into

The biggest early challenge was a silent train-inference mismatch: the training data only contained domain-name tokens, not real page content, which capped accuracy at ~59% despite looking fine on paper. Fixing it meant rebuilding the dataset from scratch — scraping real page content, sourcing verified Indian URLs, and augmenting underrepresented categories like Adult — and then catching a second, subtler bug: data leakage from repeated seed words that was inflating validation scores. On the infra side, I hit middleware auth conflicts between the RapidAPI proxy secret and direct API keys, bot-challenge detection gaps that caused silent scrape failures, and repeated hosting migrations after Hugging Face Spaces suspended Playwright usage. Every one of these required actually diagnosing root causes rather than patching symptoms.

Accomplishments that we're proud of

Getting from a 59% accuracy ceiling to an honest, leak-free 82.2% test accuracy is the accomplishment I'm most proud of — it came from real debugging, not tuning. I'm also proud of shipping this as a genuinely monetizable product: live on RapidAPI with tiered pricing, a working Chrome extension for testing, and a resilient two-microservice architecture that gracefully falls back when scraping is blocked.

What we learned

Model accuracy is only as good as the honesty of your data pipeline — the two biggest wins in this project came from fixing data problems, not model architecture. I also learned a lot about production ML realities: memory constraints forced me to cut LIME explainability from the deployed API to stay within free-tier limits, and every hosting migration taught me more about designing for portability than the last.

What's next for Website Category Classifier

Completing the Cloud Run migration for both microservices, updating monitoring to the new endpoints, and cross-listing on APILayer for broader reach. Beyond infra, next steps are direct outreach to ad networks, brand-safety platforms, and parental-control apps as design partners, plus exploring multi-label classification for websites that span more than one category.

Built With

Share this project:

Updates