Actor-Detector

Improving production quality of the movies by implementing artificial intelligence to smooth out camera movements

Comment

Inspiration

I've been dabbling around Artificial Intelligence for quite a while, especially in the field of computer vision. One of the challenges in making movies is the camera angles and how to make sure the pans are as smooth as possible. Now I realize this is not an issue with big production films but for amateurs, it really effects the production quality.

What it does

I've created a face detector program which tracks the faces of the people in the camera view in real time. This software can plug into any existing camera mount and help the motor to smoothly pan the video instead of manually doing it.

How we built it

This software was built using the open source computer vision library, OpenCV. I made use of the Haar Cascade Classifier algorithms in OpenCV which are commonly used for object detection. I then trained the program to detect human faces, but also had to use pre-trained neural network for the reasons I'll describe in the challenges section.

Challenges we ran into

One of the biggest challenges in this project was the use of OpenCV. I had no idea how to use this library and had to learn it from scratch, which was pretty cool as I got to learn something really powerful. Standard facial recognition with ample of light worked well when I trained it using a publicly available database on Kaggle, but it struggled in low light. Now that might also be because of my crappy webcam, but still it wasn't ideal since not all scenes in the movie are shot in light. I needed to make the software somewhat robust. To deal with this, I turned all the frames in grayscale and bumped up the contrast. The software detects object (in this case, the human faces) frame by frame in real time. By converting the image to grayscale and bumping up the contrast a bit, the onboard neural network was able to detect the edges better. There was a noticeable difference as it worked way better in low light now. There were still misses here and there and I attribute it to my crappy webcam which I used for testing. Another issue I faced that it was detecting the faces at their side angle accurately. I realized that's probably because it's hard to train the network on the faces which are tilted to their sides, or at least I was having issue with it since I have an M1 Mac and PyTorch hasn't released the GPU support for ARM (sad ik :( ). To counteract this, I had to make use of pre-trained weights of the neural network, which was already trained on the different angles and lighting on the humans face. This not only improved the face detection, but also worked even better in the low-light shots.

Accomplishments that we're proud of

Machine and Deep Learning is such a theoretically intensive area of study, it is difficult to train and implement the neural networks to make something like a face detector, so I'm very pleased that I was successfully able to implement NNs to detect faces. In retrospect, I'm surprised how I was able to solve the issue of detecting faces in low light. Directly implementing the concepts Andrew Ng taught in the Deep Learning specialization gives me a satisfaction like never before. I'd like to thank the organizers and MLH for organizing this hackathon. This has been really fun!

What we learned

OpenCV: A powerful, open-source, computer vision library. Earlier, I'd only heard of OpenCV and never really used it. Thanks to this hackathon, I was able to learn OpenCV and make a project using it. I can't wait to make more cool stuff with it. Also learned how to tweak the incoming data to fit the personalized needs of the project and how to use a pre-trained model to accomplish our task. This was my first implementation of Transfer Learning and I'm glad I got to make something cool!

What's next for Actor-Detector

I think the dataset I used for the training was not big enough, so more training with more random data. Although the low-light face detection works fairly well now, it still needs more tweaking and I feel it can be improved using a separate infra red camera. Lastly, installing this software on a camera mount using their API so it can be used directly by the directors and the cameramen. After working on the documentation part of it, the software can be released as an open-source for the movie directors and producers to use all across the globe.

Built With

Share this project:

Updates