The story is as follows:
Last week, I got my Iphone stolen from my class. So, I went to the Surveillance center to see the CCTV footage. It took too long to figure out the exact moment (timestamp) in the video, when the theft happened. And then the hackathon happened! You'll get an idea of what it does:
What it does
Basically, we can search within a video to find a moment! How fantastic is that! It generates descriptions of all IMPORTANT frames in the videos, and fetches the moment you search using some natural language processing. Also, using this we can organize similar videos, and remove ads from videos! cool, right?
How we built it
First, we split the video into smaller scenes by creating an average brightness histogram and calculating entropy. Scenes were distinguished and determined by substantially different histogram and entropy results. Once scenes were separated, we sent two frames from each scene to Microsoft's Cognitive services to get a highly contextual description of the scene. Iterating through each description and clustering keywords into a giant bucket, we were able to provide the users a platform to navigate through a media content with accuracy and ease.
Challenges we ran into
We tried performing scene detection using PySceneDetect library. It had no documentation. So, we had to develop our own algorithm to detect scenes.
Heavy frame computation made pages irresponsive
Accomplishments that we're proud of
My own algorithm to detect scenes! Flask Integration! Pretty much everything!
What we learned
Everything I didn't know!
What's next for Deep Search - Crime Mystery Exposed!
A Web App! Wait for it!