Training RAG Chatbot on Website Data

Inspiration

I am building customer RAG based chatbot for a client, and found that there is no easy solution to extract data from websites which is free to use.

What it does

Allow you to discover pages, identify and score them if they will fit for RAG extraction, and finally download the pages in various formats for RAG consumption, till 10th level (full website), basis requirement.

How we built it

Built it step by step by prompting in bolt.new

Challenges we ran into

Had to reset the project twice when it was not going in the right direction.

Accomplishments that we're proud of

Allows unlimited pages download now, since after each page it clears the memory.

What we learned

Future is going to change if so many paid apps can be built for nominal costs of tools like Bolt.

What's next for Training RAG Chatbot on Website Data

API support, enterprise support, login/authentication, pricing

Built With

• codetabs.com
• languages:-typescript-and-javascript.-frameworks/libraries:-react:-for-building-the-user-interface.-tailwind-css:-for-utility-first-styling.-lucide-react:-for-icons.-dexie.js:-a-wrapper-for-indexeddb
• netlify
• vite