search.io

Team pic xD

Inspiration

Let's be honest --we've all watched hour long videos because we wanted to see some particular clip in the video itself. It may be searching for some important concept in a lecture video or you just want to recollect a memory from your favourite vlogger's long videos.

What if we told you that we built a AI-powered search engine that will allow you to search exact moments in a video either based on pattern matching or contextual meaning.

Say a hearty :wave to search.io

What it does

search.io uses Natural Language Processing and mathematical algorithms to search and retrieve parts in any video based on pattern matches or the semantic meaning of your search query. The platform takes in a video URL, downloads it in the backend, performs text and audio cleaning, runs ML inference on this processed data, and retrieves the query part of the video as a timestamp. It then automatically skips to the most likely match giving you a seamless experience. Since it is a Chrome extension, it is extremely efficient and as non-intrusive as possible giving you the best viewing experience.

How we built it

System Architecture Our frontend was built using Javascript and our backend server is orchestrated using FastAPI. Python was mainly used in the backend.

The backend forms the bulk of the entire application – we perform video processing and Machine Learning inference on the preprocessed audio-visual data. After downloading the video, we either use a provided auto-generated transcript or use Google Cloud Platform's Web Speech API to get a transcript. Based on whether the user has chosen to do a fuzzy search or semantic search, we compute either the Levenshtein distances or distances from word vectors by Word2Vec. Then the shortest distance vectors are obtained along with their timestamps from the video metadata.

The frontend is a Chrome Extension written in Javascript for video manipulation. All this retrieved data are sent to the frontend for further display.

The above diagram presents a high-level overview of our system design and various tech stack components.

Challenges we ran into

FRONTEND We had to find hacky methods to ensure low latency, near real-time retrieval, and smooth user experience.

BACKEND Wrangling the transcript was a bit of a pain as it includes metadata. The biggest challenge was to obtain meaningful predictions for the semantic search. We had to explore with various corpus as people usually watch a wide range of videos, our corpus must be able to handle a wide range of vocabulary.

MISC Anything that says it's "pretty" (cough, Pretty Printing, cough) will always get you into trouble. Trust us, we learned the hard way.

Accomplishments that we're proud of

We promise it works on LONNNGG (>1 hour) videos, even with those videos, we were able to cut down processing time to ~15-20 seconds.
Text processing was a pain that we managed to get right during the wee hours of the morning. Optimising the predictions to get meaningful search results was hard due to range of vocabulary used.
Caching the predictions, text, and models allowed us to improve performance by a large margin without compromising on speed and quality.
Our MacBooks have now turned into hot plates good enough to make sandwiches on!

What we learned

With a team of ML and Full Stack devs, we thought this would be a simple enough hack to accomplish in 24 hours. But had a few humps along the road. An important aspect we learnt is how to manage expectations during a fast paced event like a hackathon. We also learnt how to build Chrome Extensions and integrate it with a ML backend. Since our hack has a lot of moving parts, integrating them was a lot tougher than we realised. But nonetheless, it was a fun ride to be on and we learnt a lot from this experience.

What's next for search.io

Move to a completely serverless environment
Distributed Processing for extremely quick search results
Support for any HTML5 video, irrespective of platform (YES, we are looking at you NTULearn & LumiNUS)
Improved semantic searching with support for pretrained vectors