We wanted to build something that allows you to search for things in videos just like you do in normal files and summarise long texts according to the available time you have. Since we had a good number of people in our team, we distributed the work in such a way that it allowed us ti build not only this but a lot more.
What it does
It allows users to search for things in videos just like you would search for stuff in text files and highlight the appearances of the searched keyword in video's timeline and allows you to skip directly to that part. It also allows you to summarize long texts according to your reading habits and the available time you have. We also built a cross platform chat system along with video calling to support the overall functionality with a feature to broadcast your mobile screen to larger devices through socket connections.
How we built it
Used MEAN stack for the development of the entire project. The text summarizer, chat system, video calling and video search were built on Node.js (The only real dev language) with MongoDB as the back end. Socket.io was used to establish socket connections between the devices for broadcasting mobile screen sizes to larger screens. HPE's Audio to text API was used to convert video files into text for videos without captions to search for tags for highlighting on the timeline. Finally built a clickable wireframe using Socket.io for turning these stand alone things into a working app model.
Challenges we ran into
Couldn't find tag creation in Youtube's API for tagging in timelines, which was kinda disappointing. Had to do it the old fashioned way by converting the audio into text, mapping the audio with screen time and then searching and highighting it on the timeline. n
Accomplishments that we're proud of
Everything that we thought we'd build, we built. Got the first overall position at the hackathon and got a lot of media attention since we'll represent our country, India in the global finals.
What we learned
That proto.io works like a charm with it's interactions panel and Node.js is capable of anything and everything.
What's next for sudo
Shout out to all open source enthusiasts for tweaking and contributing. Going to open source it.