Inspiration
Help law enforcement and other agencies find and prevent human trafficking with ML and date curating
What it does
Web scrapes postings on sites like Craigslist and identifies which ads are related to sex work
How we built it
Built a web crawler to collect data, built an algorithm to convert data for ML input, built an ML classifier using a naive bayes algorithm
Challenges we ran into
Finding the data and generally completing everything in time
Accomplishments that we're proud of
Being able to develop a prototype and get everything set up in the short 24 hr period.
What we learned
Distinguishing between consensual sex work and human trafficking ads is very difficult, especially without a proper data base - lead us to just distinguish between sex work and non sex work ads for now as a proof of concept. I also personally learned more about ML and python programming. My team mate learned that python indexing can be a pain the ***.
What's next for Human Trafficking ML algorithm
Further train the algorithm and expand our data set to include flagged trafficking posts versus consensual sex work to distinguish between the two and further help law enforcement and other agencies dedicated to preventing human trafficking
Data
Training data was obtained from data.world (backpage escort ads) and craigslist (web-scraped for non-escort ads)
Libraries
The Craigslist web scraper requires a pip install of scrapy to run with the .py file provided. The conversion of the web-scraped excel sheet for ML input requires a pip install of pandas to run with the .py file provided. The ML algorithm requires a pip install of sklearn to run with the .py file provided.
Commands to run from command prompt
Web Scrape - scrapy crawl h4hCrServ -o h4h_data_2.csv Conversion to ML input - python data_parsing.py ML algorithm - python ConvertDataAndML.py
Notes
Web scrape Craigslist - Exact craigslist website to scrape can be changed in the h4hCrServ.py file by changing the start_urls variable