Veridex

Inspiration

Misinformation spreads faster than fact-checks can keep up. We wanted to build a tool that doesn't just search the web -- it verifies what it finds. Every claim traced to its source, every source scored for credibility, every conflict between sources surfaced. Think of it as: what if Perplexity showed its work?

What it does

Veridex is an autonomous web research agent that:

• Searches the open web via DuckDuckGo, retrieving 10+ sources per query with automatic deduplication and relevance filtering
• Scrapes pages responsibly with robots.txt compliance and configurable rate limiting using BeautifulSoup + lxml
• Extracts structured facts using spaCy NLP -- named entities (people, orgs, dates, locations), key phrases, statistical claims, and factual assertions
• Scores source credibility across multiple dimensions: domain authority, HTTPS usage, content depth, citation density, and known-source matching, displayed as radar charts
• Cross-verifies claims across all sources to identify consensus points and conflicts, shown as a claim verification matrix
• Generates exportable reports in Markdown, PDF, or JSON with full citations, entity tables, and credibility breakdowns
• Persists research history in SQLite for search, filter, and re-run of past queries

How we built it

The pipeline flows: Query -> DuckDuckGo search -> Web scraping (robots.txt aware) -> spaCy NLP extraction -> TF-IDF TextRank summarization -> Credibility scoring -> Cross-source verification -> Report generation -> SQLite storage.

The frontend is a multi-page Streamlit dashboard with five Plotly analytics charts. The backend uses classical NLP throughout -- no LLMs, no API keys, no hallucinations. All intelligence runs locally.

An offline demo mode with three pre-cached research queries (AI in healthcare, quantum computing, climate/renewables) works without network access.

Challenges we ran into

Building a credibility scoring system that's meaningful without access to proprietary databases was tricky. We settled on a multi-dimensional approach combining domain trust lists, HTTPS verification, content depth analysis, and citation density metrics. Getting cross-source verification to reliably detect consensus vs. conflict required careful TF-IDF similarity thresholding.

What we learned

Classical NLP techniques (TF-IDF, TextRank, NER) are surprisingly effective for structured fact extraction when you don't need generative capabilities. Responsible scraping with robots.txt compliance and rate limiting is both ethical and practical -- it prevents IP bans.

What's next

Adding claim temporal tracking (how claims evolve over time), integrating academic paper sources via Semantic Scholar API, and building a browser extension for real-time fact-checking while browsing.

Built With

• beautiful-soup
• python
• scikit-learn
• spacy
• sqlite
• streamlit