Usually start watching an already started streaming is challenging. You don't know about the previous joke that the streamer has told, you don't know about the previous highlights that the streamer made in the game that he/she is playing... In conclusion, the experience become less inmersive and it's harder for the audience to not leaving the stream.
In this context, in which retaining as much audience as possible is necessary in order to have good Ads incomes, our project tries to fill this gap by creating a summary with the previous important moments of the streaming, both manual and automatically
What it does
The project is based on two sides: the viewer side and the streamer side.
For the viewer the things are simple: when he/she enters to the stream, a video showing the past highlights is showed before the current streaming.
For the streamer side, we tried that make things simple. The streamer window have a display of the current highlights recorded for the system (both manually and automatically). Manual highlights can be recorded by using the streamer interface. Automatic highlights are decided based on a combination of streaming frames and a combination of keyboard and mouse input analysis, in which the sections with more impact are recorded.
The most interesting highlights, based on a internal ranking that score the impact of each one, are displayed to the user in a clip no longer than 2 minutes
How we built it
The web application is build with React.js to the frontend and Django and Python to the backend. In addition, we developed Python modules to process the streaming frames and a C++ binding to Python to keep track of keyboard and mouse events (Logitech SDK didn't worked in our pcs).
The communication between the backend and the different modules is done by using a database as a middleware, in which all the relevant information is stored.
In addition, the automatic clips are selected by using different kind of features (keyboard, mouse, frames) as an input to a model, which outputs the score of groups of frames. Later, by analysing the time series generated, we can obtain the most interesting parts of the stream.
Challenges we ran into
Logitech SDK doesn't work on our pcs, so we had to make drivers to read both mouse and keyboard.
In addition, our database system was not robust enough so at the moment of merging the different components of the project we had a lot of problems to integrate the different technologies.
Finally, MacOsX is always a problem with like EVERY LIBRARY, so we ran into problems while compiling the parts of the code made with C++
Accomplishments that we're proud of
We have a system capable to extract the most interesting parts of an streaming based on streamer performance on games. In addition, this is the first time in which we have an interface, so this is a great achievement for us.
What we learned
We learned about why using a debugger is one of the best things to do in order to improve programming performance. In addition, we learned about reading devices with Linux and a little bit of web development.
What's next for hAIghlights
The first thing to do is clean the code and change the database. Later, a good way to go would be integrate the application in OBS and StreamLabs. Finally, would be a good idea to improve AI performance by doing sentiment analysis on the Twitch chat, in order to obtain the moments in which the audience is more excited.