There are massive data streams out there in the whole internet, each with valuable information attached to it. What if a photo of a girl leaned against house no. 123 with shade of street name X , might actually be a victim of human trafficking and with timely manner action, this photo can be a survival point. Traditionally, with manual investigation, it takes weeks to find the location of photo based on the texts and more importantly it needs exhaustive search to find the identity of the victim as photoed. with the emerge of social media and Hashtag, everybody posts photos anywhere and tag it with any desirable Hashtag. with those Hashtags refering to a criminal case, there is a need for an AI tool which can parse a tagged photo and detect a face, a car and/or location. What we deploy helps law enforcement, government and the private sector identify and combat criminal activities in a matter of seconds compared to days. Tag the photo with #RouteToRescue and our tool will do its job.
RouteToRescue is an AI tool which not only tackle with human trafficking and criminal cases but also visualize unstructured data to easily search, track and index data for retrieval.
What it does
RouteToRescue relies on social media power to help human trafficking. Social Media hashtag will be the greatest source of information as social media users can tweet, send post or share any suspicious activities and tag them by relevant hashtags. For example, a social media user posts a suspicious activity, take a picture of the scene and tag it with proper hashtag such as #RouteToRescue or #humantrafficking. This information is very rich and might include many valuable information with good amount of details for example face of a victim or trafficker, a location details, building information or a vehicle model, color or license plate. Also, this shared posts has some metadata such as actual time/date of the event as well as location of the event (if possible to share).
RouteToRescue application crawls through all relevant hashtags and parses the data associated with it. Analyze the images via powerful image processing techniques using AWS Rekognition service to detect objects, activities, faces, sentiments, texts, vehicle models, license plate, scene details etc. These enriched attributes will save on the systems database and will be available for other components of the system. In first step, RouteToRescue applies several filtering techniques such as text semantics and face detection and analysis (detect attributes such as gender, emotion, age range, eyes open, glasses, facial hair for each) to make sure social media post is related to human trafficking. After filtering, each filtered item will be saved on the database. RouteToRecuse has a capability to detected a face in each saved item and cross-validate detected face with external databases such as NamUS and National Crime Record Bureau as well as historical data recorded by RouteToRescue itself. This capability provides fast and accurate face search to identify a person in a photo or video. Also, it can verify identity by analyzing a face image against historical images stored in the system for comparison. RouteToRecuse has capabilities such as unsafe content detection, object detection, vehicle detection (model, color, license plate) and text detection to read building numbers and scene details.
After image analysis if RouteToRecuse identifies a victim or a trafficker, it will send notification to registered authorities such as police departments in the closest zip code of the incident. Moreover, RouteToRecuse uses MapBox APIs to create a heat-map and tracking path of the similar incidents in different location and time lines. For example, if a detected vehicle or person has been reported in several other posts and locations, the visualization tools shows the timeline and map locations of all similar reported person or vehicle. This will be a great tool for authorities to track the person and identify more details about their network.
How I built it
RouteToRecuse has been built on top of Amazon Web Services. For ingesting the data from Social media, a fleet of virtual machines (AWS EC2) crawl the list of predefined hashtags form Facebook, Twitter and Instagram to stream the images, videos and text to Amazon Simple Storage Service (S3) via AWS Kinesis Stream. Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance. Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming service. KDS can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. After the images landed on S3, S3 will trigger a Serverless AWS Lambda function to apply image analysis, face detection, object detection, sentiments analysis as well as License plate detection via AWS Rekognition service. Amazon Rekognition can identify objects, people, text, scenes, and activities in images and videos, as well as detect any inappropriate content. Amazon Rekognition also provides highly accurate facial analysis and facial search capabilities that you can use to detect, analyze, and compare faces for a wide variety of user verification, people counting, and public safety use cases.
The output of all this analysis will be sent to AWS NoSQL DynamoDB database. For each saved item another Lambda function analyzes the data with external missing person datasets such as NamUS as well as saved historical data in the system. For each high confident output a notification will send out to down stream systems such as Police departments or Human Augmented system for further analyze the detected cases. Also all the detected items will be plotted on MapBox to create a holistic tracking system to track each case over time and location. Also all the data will be sent to Elasticsearch for indexing and searching. Elasticsearch is a distributed search and analytics engine that allow easily search in million of documents and addressing a growing number of use cases. With this tool we can easily extract the location as well as timeline of all related cases and build alerting and monitoring systems for each high profile case.
Challenges I ran into
Getting the authorized dataset, which we solved by UI FBI and Interpol missing person database. How to create a heatmap and locate all the images, we solved this bu Integration with MapBox
Accomplishments that I'm proud of
We are so proud building the end to end system in a very short time and provide a fully automated solution for such a critical problem.
What I learned
During this hack I had a chance to learn a lot about human trafficking by reading all the articles posted on the Hack Slack channel as well as listening to WebEx panel presentations and get more deeper understanding about the problem.
What's next for RouteToRescue
Next step for RouteToRescue is building the solution on scale and make it available for public use.
To be integrated with Real Camera on the streets and public/private places and do parses when any captured photo/scene contains shocked/fear face. as soon as it hits a match, authorities will be notified.