Auto Video Highlighting

• 

  Auto-generated video highlight (see how it corresponds with emotion analysis!)

• 

  Live comment fluctuation, emotion and sentiment analysis.

Definition: Live comments are comments integrated onto the screen of video or live streaming. These comments are associated with a particular time of a video, and provide a more vibrant experience of video watching and online community building.

Inspiration

My team are all fans of bilibili, a video platform known for its community of amateur video producers and live commenting. We realize that there are much to explore in these comments indexed to specific time of video, and we want to use that information to bridge the gap between video makers and viewers by providing better video previews and comments analysis.

What it does

It automatically generates attracting highlights and previews for videos and live streaming with live comments, and provides fluctuation and emotion analysis to video producers to better their work.

How we built it

We trained 2 bidirectional LSTM networks for sentiment analysis and emotion classification in the context of Chinese internet slang. We scraped reviews on bilibili.com as data for sentiment analysis. We designed a video assignment algorithms that outputs highlights of a video. We also built a data visualization interface that displays the above data and highlights.

Challenges we ran into

A big challenge we encountered is the difficulty of analyzing internet slang. Available machine learning APIs trained on formal text have bad performance on live comments, so we have to pick and train our model from scratch. Luckily there are many online resources and people at cal hacks to help, and we eventually achieved a performance much better than existing APIs.

Accomplishments that we're proud of

Being able to output highlights that are really close to human judgement!

What we learned

As a team with minor prior experience in Natural Language Processing, we managed to quickly pick up skills that we need and apply them in making a real project. We also have more insights into the relationship between videos and comments. Moreover, we realized that building a project is fun and not as hard as we thought.

What's next for calhacks-danmuku-public

We want to expand its impact in real applications to help video makers and viewers better understand each other. We also look forward to integrating it with video and audio analysis for more interesting results.

Built With

• css
• google-cloud
• html
• javascript
• jupyter-notebook
• keras
• python