CTRL-F

Home page
Results page

Inspiration

The amount of data in the world today is mind-boggling. We are generating 2.5 quintillion bytes of data every day at our current pace, but the pace is only accelerating with the growth of IoT.

We felt that the world was missing a smart find-feature for videos. To unlock heaps of important data from videos, we decided on implementing an innovative and accessible solution to give everyone the ability to access important and relevant data from videos.

What it does

CTRL-F is a web application implementing computer vision and natural-language-processing to determine the most relevant parts of a video based on keyword search and automatically produce accurate transcripts with punctuation.

How we built it

We leveraged the MEVN stack (MongoDB, Express.js, Vue.js, and Node.js) as our development framework, integrated multiple machine learning/artificial intelligence techniques provided by industry leaders shaped by our own neural networks and algorithms to provide the most efficient and accurate solutions.

We perform key-word matching and search result ranking with results from both speech-to-text and computer vision analysis. To produce accurate and realistic transcripts, we used natural-language-processing to produce phrases with accurate punctuation.

We used Vue to create our front-end and MongoDB to host our database. We implemented both IBM Watson's speech-to-text API and Google's Computer Vision API along with our own algorithms to perform solid key-word matching.

Challenges we ran into

Trying to implement both Watson's API and Google's Computer Vision API proved to have many challenges. We originally wanted to host our project on Google Cloud's platform, but with many barriers that we ran into, we decided to create a RESTful API instead.

Due to the number of new technologies that we were figuring out, it caused us to face sleep deprivation. However, staying up for way longer than you're supposed to be is the best way to increase your rate of errors and bugs.

Accomplishments that we're proud of

Implementation of natural-language-processing to automatically determine punctuation between words.
Utilizing both computer vision and speech-to-text technologies along with our own rank-matching system to determine the most relevant parts of the video.

What we learned

Learning a new development framework a few hours before a submission deadline is not the best decision to make.
Having a set scope and specification early-on in the project was beneficial to our team.

What's next for CTRL-F

Expansion of the product into many other uses (professional education, automate information extraction, cooking videos, and implementations are endless)
The launch of a new mobile application
Implementation of a Machine Learning model to let CTRL-F learn from its correct/incorrect predictions

Built With

computer-vision
express.js
github
google-vision
ibm-watson
machine-learning
mongodb
natural-language-processing
numpy
python
recurrent-neural-networks
sql
sqlalchemy
vue

Submitted to

QHacks 2019
- Winner QHacks - Overall Winning Teams
- Winner YC - Top 5 Overall Teams
- Winner Ramen Ventures- Top 5 Teams

Created by

I primarily worked on the research and testing to discover the best and most compatible technologies (whether it be APIs or neural networks) to integrate in our project. Moreover, I worked on the smooth integration of all the different parts of our project.

Richard Xie
I built the front-end using Vue.js, set up our MongoDB database, and developed a RESTful API to allow communication between the two in order to display our processed data to the user.

Brittany Lau
I worked on the speech-to-text functionality by using Python to extract audio from video files and returning detailed transcripts. I also helped design the database and integrate it with the rest of the application.

Jayson Lou
Worked primarily on the computer vision aspect of the project using python. Created functionality to extract frames from videos and retrieve key information from the frames for future searches.

Stephen-Choi
Rahib Kanjiyani