Inspiration
As Cyber Security Analysts, throughout our day we consistently access the dark web for information regarding clientele. DarkSherlock aims to automate and allow for ease of access to specified forums through the Tor Browser. Throughout the course of our internships we access these forums multiple times a day often getting signed out after a mere fifteen minutes.
What it does
DarkSherlock allows us to automatically login and search for specific users / vendors and screenshot the specified users profile.
How we built it
This project was mainly built with Python, Selenium, and, of course, the Tor network. We had to connect to the Tor network to be able to view websites with the .onion domain. Using Selenium the program opens the Tor Browser, connects to the website, then takes a screenshot of the user’s information.
Challenges we ran into
Working with dark websites created many issues with the project. To start, opening the Tor web browser using Python is more difficult than it sounds. CAPTCHA’s are also a big part of the dark web, and since we were trying to get a bot to access the website, we needed human interaction. Thankfully there were ways to work around this issue, by opening the browser for the user to complete the CAPTCHA then continuing the bot once everything is completed solved the problem. The last problem was websites detecting our bot. Dark websites try to do everything they can to deter bots from entering the website. For some of the websites we needed to visit, we needed to add more protections to our web browser to evade detection.
Accomplishments that we're proud of
We are very proud to say that it is possible to make a bot for the dark web. Though there are many CAPTCHA’s and bot detection methods, we were able to automatically connect to and scrape dark websites. After showing this tool to professionals in the Threat Intelligence field, they stated that, with a little extra polish, this program would be easily one of the best tools for dark web researchers.
What we learned
Programming a web bot for the dark web is much more difficult than a normal one, and is most likely the reason this was not created before. There are a lot of intricacies to work with when finding and scraping hidden websites. We learned a lot about how the Tor web browser works, what websites are on the dark web, and how cybercriminals communicate with each other through forums.
What's next for DarkWebSherlock
With the feedback we received from the Threat Intelligence community, we will continue development on DarkWebSherlock. The amount of websites DarkWebSherlock currently has access to is one of the biggest improvements we could make. In addition, we want to scrape more information from the profiles we can access with DarkWebSherlock.
Log in or sign up for Devpost to join the conversation.