Inspiration

Since last December, I have been selling enterprise software to companies like Molson Coors and Getaround to help them make their offline marketing more efficient (at supermarkets, restaurants, bars, events, field marketing, etc). The technology we've been selling them is a mobile software solution that allows their merchandisers, field reps and brand ambassadors to report marketing data from the field directly to their marketing manager's slick analytics dashboard in real time. This type of field data includes supermarket audits, consumer surveys, event reports and that sort of thing.

A month or so ago, we came to the realization that these Brand Ambassadors who push their company's products at events or at points of sale are the only people in the entire distribution chain of these large multinationals who interact with the company's end consumers on a recurring basis. This is especially true for Consumer Goods companies. We looked at some numbers that our clients were reporting with our tool, and we realized that for each individual brand, these Brand Ambassadors are talking with more than 2 Million individual people per month all over the world on aggregate. We also realized the quality of these conversations was incredible, given that Brand Ambassadors talk to consumers at the places where the consumers would typically buy their product. We realized that if we could manage to capture insights from those millions of high value conversations, we would give these type of companies the opportunity to understand their consumers, their market, and their selling points better than they have ever been able to before.

At around the same time, my friends from high school Chris, Oscar, and Michael were working on an audio recognition algorithm that would recommend songs based on the features of the actual sound of the music. Their audio tech was so awesome that Michael got a call from Tim Cook and the head of engineering at Apple who wanted to learn more about his tech to implement it for iTunes. The idea for Rilla came as a natural inspiration from our many conversations.

What it does

From the Brand Ambassadors' phone, our audio recognition will activate when they are at an event, and listen to the conversations they have with consumers about the brand on the field. It'll capture insights like the gender of the consumers, their level of brand awareness, their willingness to pay for the product, and keywords that might indicate why they buy the brand or not. Similar to Auravision, there's no need to install expensive and complicated hardware, or change the workflow of these companies at all. They just keep doing what they were doing, sending their BAs out there, they go with their smart devices on hand, and our tech does the rest.

How we built it

We built a front end structure in React that shows the dashboards and the captured insights as well as the recording interface. Then we built a Python backend where we would store the audio file of the recording and run all the voice recognition and natural language processes on. Since we had less than 24 hours, we had to rely on public models that had already been trained before, and on public APIs like Google's voice diarization API, and their Speech to Text API. We managed to build models that capture the gender of the consumer and the frequency with which they buy the product.

Challenges we ran into

Building anything that involves machine learning models in less than 24 hours is incredibly difficult, but the most difficult part for us was integrating all our systems into a singular MVP that wouldn't crash.

Accomplishments that we're proud of

We didn't sleep, we were jetlagged, and we barely finished... but we finished.

What we learned

Audio recognition is a very new field that's only recently beginning to gain traction, so we're going to have to do a lot of the research and development for ourselves. We are going to have to train our models with the data that our clients collect over time, and are going to be having to research things for our particular industry that not a lot of people have researched before.

What's next for Rilla - Audio Recognition for offline marketing

We will continue building on what we did here. Right now we can only capture the gender and the type of costumer that the BA talked with, eventually we want to be able to capture things like a consumers willingness to pay for a product, use all that marketing data to predict sales more efficiently, and expand our stronghold in the industry with our own proprietary models.