ResearchForge

A 4-agent research pipeline that searches the web, scrapes the most relevant source, writes a structured report, and critiques its own output — with a Streamlit UI on top.

Built with LangChain, Groq (Llama 3.3 70B), and Tavily.

How it works

The pipeline runs four stages in sequence, passing state from one to the next:

1. Search Agent — uses Tavily to find recent, reliable sources for the topic and returns titles, URLs, and snippets.
2. Reader Agent — picks the most relevant URL from the search results and scrapes clean text content from it using BeautifulSoup.
3. Writer Chain — combines the search results and scraped content into a structured report (Introduction, Key Findings, Conclusion, Sources).
4. Critic Chain — reviews the report and returns a score out of 10, strengths, areas to improve, and a one-line verdict.

Project structure

.
├── app.py              # Streamlit UI
├── pipeline.py         # Orchestrates the 4-agent pipeline
├── agents.py           # Agent/chain definitions (search, scraper, writer, critic)
├── tools.py            # web_search and scrape_url tools
├── requirements.txt
├── pyproject.toml
└── uv.lock

Setup

1. Install dependencies

pip install -r requirements.txt

or, if using uv:

uv sync

2. Set environment variables

Create a .env file in the project root:

GROQ_API_KEY=your_groq_api_key
TAVILY_API_KEY=your_tavily_api_key

• Get a Groq API key from console.groq.com
• Get a Tavily API key from tavily.com

Usage

Run via terminal

python pipeline.py

You'll be prompted to enter a research topic, and the pipeline will print progress and results for each stage.

Run via Streamlit UI

streamlit run app.py

Enter a topic, hit Run Pipeline, and view the report, critic feedback, raw search results, and scraped content in separate tabs. The report can be downloaded as a .md file.

Tech stack

Notes / limitations

• The reader agent only scrapes one URL (the one it judges most relevant from the search results) — it doesn't cross-reference multiple sources.
• Scraped content is truncated to 3000 characters per page to keep context manageable.
• The Streamlit UI currently shows a single loading state for the full run rather than live per-stage progress, since pipeline.py returns only after all four stages complete.
• No retry/error handling around LLM or API rate limits yet — long-running or repeated queries may hit Groq/Tavily limits.

Possible next steps

• Stream per-stage progress into the UI (convert run_research_pipeline into a generator or add callbacks)
• Let the reader agent scrape and synthesize multiple sources instead of one
• Add report export as PDF/DOCX
• Cache search/scrape results to avoid redundant API calls on re-runs

Built With

• python