Deepfakes are hyper-realistic synthetic media developed using Artificial Intelligence, for various purposes- but mainly malicious purposes, like undermining journalism and people's trust in democratic government institutions, harming the reputation of people (eg: revenge pornographic deepfakes), and spreading fake information to widen gaps in societies to cause riots, etc. Deepfakes are rising at a rate of 6280 times (Year-On-Year) and are reaching people 6 times faster than real news. This is a growing threat and thus it is important to counter them by developing tools that can help us detect them, and prevent them from spreading. This inspired us to work towards solving the problem, and so we developed Detectd.
What it does
Detectd uses the same technology to detect deepfakes that are used to create them - Artificial Intelligence. Detectd allows the users to upload media- video or image to our web platform and verify the authenticity of media, instantly. This will help people and institutions verify the authenticity of media that they consume or share with the public, and help counter the malicious actors who rely on disinformation and weaponization of AI for criminal activities.
How we built it
We used Python, its multiple libraries, and Tensorflow framework to develop a Convolutional Neural Network combined with Long Short Term Memory Network to analyze 40 frames from a video in a sequential manner. While we used Flask to deploy the model in a WebApp and serve it to the public through Azure App Services. Our CNN was pre-trained on the ImageNet dataset and we further trained the entire model on FaceForensics++ and DFDC dataset. This has made our model more robust- it can now detect a variety of manipulated media- Deepfakes, FaceSwaps, Face2Face, and even Neural Textures.
Challenges we ran into
We ran into some challenges not while developing but deploying the project. We are using the videos uploaded by the user as input. So sending the videos from front-end to back-end in a manner that is fast, secure, and efficient we used the Base64 encoding. Also, one of the libraries we are using is OpenCV and the deepfake detection model is developed using Tensorflow. So while deploying the project to Azure App Services, we faced trouble due to a specific library- LibGL.so.1, which was required for OpenCV to function. The problem did not arise in our local environment (Windows) but the app deployment container (Linux). However, we solved this using SSH- we installed the libraries manually, remotely, and also edited our code to ensure that it works every time. Another issue we faced while deploying the model was that the model which was stored in the TF SavedModel format could not be loaded and hence could not be used. We solved this error by making changes to our code that force loads the model at startup, before the first user request.
Accomplishments that we're proud of
Multi-tasking. We implemented a complicated Convolutional Neural Network - Long Short Term Memory Network within 4 hours from a research paper. Then we trained it on high-quality deepfake data and ultimately deployed the model to Azure App Services. But we are most proud of deploying our model for public use. Since this was our first time deploying a model. Also, we’re the most accessible deepfake detection platform capable of video and image deepfake detection.
What we learned
We learned how to use Flask and Microsoft Azure App Services for deploying a deep learning model for public use, on the web. We also learned a lot about the functioning and implementation of LSTM and using Azure Bot Services to create a chatbot.
What's next for Detectd
We plan to add the functionality to detect deepfake audio and provide deepfake detection through a smartphone app. Post this, we plan to launch Detectd as a startup.