-
Tarantula
-
Tarantula in action
Inspiration
I needed a crawler, as most of the open source crawlers are complicated to use and takes effort to learn.
What it does
Provide domains, run a crawl, and browse saved results from the interactive terminal UI
How we built it
The initial plan was to go for a framework, but out of workforce, I used gems instead to get going. I did split the project into three. UI section, Worker section, Database section.
Challenges we ran into
Database management and terminal UI was a challenge. Took it, solved it!
Accomplishments that we're proud of
Easy to use crawling which cuts off most of the complications and easy to customise.
What we learned
Advanced Ruby, Crawling mechanism, Database Management.
What's next for Tarantula
Make it fully customisable and export projects as .json format to export project to - [apis]- Proxy
