Youtube recently demonetized a large pool of videos many of which were made by career content creators. They cited advertiser friendliness as the reason and targeted videos tagged or flagged with unsavory topics. This caused quite the stir in the Youtube community with many feeling snubbed by high-view content that earned piddling revenue.
We wanted to create a content creation platform where the creator would be rewarded not by the amount of money advertisers were willing to pay but by consumers engagement with the content. We wanted a system that would gauge interest and appreciation for media without unduly burdening the user. To that end, we built Aria- a platform where emotional analysis of viewers through facial and voice processing measure engagement would appropriately allocate funds to video creators.
How we built it
We developed a web application where we output a video and, during the playback of said video, we grab visual and auditory information. We send this information back and process the visual information using Microsoft's cognitive service API for emotional analysis and the auditory information using Microsofts speech processing and text analysis APIs. We store these in a documentdb for later retrieval. When calculating costs, we retrieve information from our database and combine these values together into a metric that measures the difference between content creator intent and perceived user emotion and achieve a metric for how well a video achieves its goal. Using this value, we transfer an appropriate amount of funds from a central revenue bank account to content creator accounts using the Capital One Nessie API.
Challenges we ran into
We did not anticipate the difficulty of processing audio data. Taking data in as input, sampling correctly, converting the file format, and ultimately converting this into a concrete sentiment value were all more challenging than we anticipated.
Accomplishments that we're proud of
For the bulk of the video presenting software, we remained entirely loyal to Microsoft's services and integrated each in a meaningful way.
What we learned
Sometimes simple ideas have difficult implementations. Even when we broke down our project into discrete, manageable chunks, unforeseen difficulties and misunderstandings resulted in a lot of time spent tinkering with code. Given a short time period to construct our project, such as a hackathon setting, focus on essential components is paramount.
What's next for Aria
Our video system is in a very early stage of development. Many features that are common across platforms such as Youtube, Vimeo, etc such as recommendations are absent in our current iteration. Aria can be expanded to achieve these while still retaining its core goals.