Inspiration
I came across the tool called katana by project discovery, found it very tricky to use and it takes effort to learn, to took a step to build my own.
What it does
Crawls for web pages, links, files. Then saves them in a separate space. And it is also configurable at some point.
How we built it
The app is structured as microservices software. Every component works on its own. I used Ruby as the language Ruby and Shell as executable.
Challenges we ran into
UI was a challenge as curses are a new topic to me.
Accomplishments that we're proud of
Usability and Simplicity makes tarantula best option for a web discovery tool.
What's next for Tarantula
Machine learning models to solve detected captchas to continue crawling.