ScrapeMind

• 

• 

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for ScrapeMind

Inspiration

Ask a chatbot about a web page and it will often answer confidently about things the page never said. That is fine when you are chatting. It is not fine when you are checking a number in a report or a clause in a policy.

I wanted the opposite behaviour. Read one page. Answer only from that page. If the answer is not there, say so.

What it does

You give ScrapeMind a URL and a question. It fetches the page, finds the parts most related to your question, and answers from those parts only. If the page does not cover it, it tells you that instead of guessing.

Example, from the Simple Wikipedia page on Tim Hortons:

Q: How many locations does it have?

A: I could not find the total number of locations on the page. The provided text only mentions specific historical milestones such as the 500th store opening in 1991, and notes there were 500 locations in the United States as of December 2008, but it does not give a current or total location count.

It found the related facts, and still refused to invent the number I asked for. That refusal is the feature.

How I built it

This is a retrieval augmented generation pipeline, in five steps:

1. Fetch. Load the page text with LangChain's WebBaseLoader.
2. Split. Cut it into 2000 character chunks with 200 characters of overlap.
3. Embed. Turn each chunk into a vector and store it in a local ChromaDB collection, one collection per URL, keyed by a hash of the URL.
4. Retrieve. For a question, pull the 4 closest chunks by similarity.
5. Answer. Pass only those chunks to the model, with a prompt that tells it to say it could not find something rather than guess.

Chunk IDs are a hash of the chunk text, so indexing the same page twice overwrites the same rows instead of piling up duplicates.

There are two front ends. A Streamlit app for clicking around, and a CLI for the terminal.

Challenges I ran into

No free OpenAI tier. I built it against gpt-4o-mini and then found the OpenAI API has no free tier and a $5 minimum top up. I moved the whole model layer to Google AI Studio's free tier instead. The retrieval logic did not change, only the model calls.

Answers came back as objects, not text. Gemini 3 returns reply.content as a list of content blocks rather than a string. My CLI was printing raw dictionaries with a wall of base64 in them. I wrote a small helper to unwrap the text blocks.

The free tier counts every chunk. A long Wikipedia page became 147 chunks, and the free embedding quota counts each chunk as its own request. It failed instantly. I made the chunks bigger to halve the count, then added a wrapper that sends them in groups of 25 with a pause and a backoff retry between groups.

Some pages have nothing to read. Retail sites block scrapers, and many modern sites build their content with JavaScript after the page loads, so the HTML is nearly empty. ScrapeMind reports what it actually received in those cases, which is the honest result even though it looks like a failure.

What I learned

The prompt is doing more work than I expected. Telling the model to admit when something is missing is what separates a grounded answer from a plausible sounding one, and it costs nothing to add.

I also learned that reading error messages properly is faster than guessing. Every one of the bugs above announced exactly what was wrong. The quota error even named the metric and the limit.

What's next

• Show the retrieved chunks next to the answer, so you can see where it came from
• Handle JavaScript rendered pages with a headless browser
• Support asking across several pages at once

Built With

• chromadb
• embeddings
• googlegemini
• langchain
• python
• rag
• streamlit
• vectordatabase