When you're looking for not only a video on a topic, but a specific sub-topics within a video, it can be hard to locate it. We realised that it would be very helpful to be able to search within and across videos to locate what you're after, whether that's for lectures, TED talks, or other videos.
What it does
We make online lectures searchable, without requiring lecturers to do anything. Our chrome extension lets you search for specific content within a lecture and across multiple lectures, and jump straight to that part of the video.
How I built it
We built it as a Chrome extension that embeds our own search bar within the online lectures website. We obtained the text to be searched across in two ways:
- Audio was extracted from the lectures and sent to IBM's Watson to provide us with a transcript
- Images were taken from the video stream every 60 seconds, and sent to Azure's service to extract text from within images
These two sources of text provided us with the data to use with Algolia to power the search. The Chrome extension interacts with the search API to display results, and with the DOM of the video site to show the search bar and jump to the right video and time in video.
Challenges I ran into
- Some of the libraries for react did not work as expected
- Struggled to find a good search platform until we settled with Algolia
- Difficulty processing the json responses from Watson and Azure to put it into a good format for searching across
- Differentiating between a = b and a == b while sleep deprived ;)
- Committing node_modules to repo (upside: gave one team member most-lines-changed accolade)
- Getting CSS right
- Audio transcription is not great (supplementing it with text from images helped significantly)
Accomplishments that I'm proud of
- Looks great
- It works!
- Nice finishing touches to the user experience
- Process for adding new videos will be easily automatable with the tools we've now written
- Finished within a reasonable time
What I learned
- Algolia is very fast
- Manipulating data structures
- Using React better
- Tiredness significantly reduces productivity
What's next for Searchable
- Create a business plan (if commercialising it is a reasonable path)
- Gather feedback from prospective and actual users
- Improve the search algorithm to produce more meaningful results
- Automate processing of videos