(image 1) before logging in
(image 2) after logging in
(image 3) add to playlist
(image 4) create new playlist and song to playlist
(image 5) playlist scheduler
I wanted to mention we had a some issues when migrating some stuff today. We had to render locally for the video. Also sorry for the quality and jump edits. But want to get it out! Def will be making improvements over the next few days. We'll submit now, but will be submitting updates over the course of the next few days to get it working again. Also I wrote this while the devs have been working round the clock to fix things, so there may be a few technical things that have changed or I might have missed but this is a pretty solid overview. Anywho, let's dive in!
To me, there is no greater feeling than that instantaneous spark of wonder from a new idea. These jolts are always preceded by a new piece of content that helps me re-connect and re-envision the world around in a slightly new light. This yearning for new feelings is what simultaneously propelled me into the analytical craft of applied physics and the creation of music (and now memes as well). I want nothing more than to share that rush of exciting moments and powerful content with as many people as I can.
A year before graduating Rice U, I had to decide if I would actually be able to pursue my ambitions mentioned above and figure out what that even looked like. Did I have to leave behind my passion for science for music and creative content, or vice versa? This was an even tougher question I saw entrepreneurship as an opportunity to maximize the positive impact on peoples. I decided to at least starting start studying "How do artists promote themselves?" and approached the question like my applied physics research.
After a year of reading the literature, seeing to presentations, and networking with thought leaders, I think I have a strong grip on how to help more spine-tingling moments happen (besides Twitch ASMR channels, though those are cool concepts in their own right!). The funny thing is that the answer had been in front of me. I had been slowly getting into gaming and Twitch. There was a ton of music being played right there and I would have done anything to get my music played by streamer.
After talking to other artists/labels, and streamers. I realized streamers have a similar but inverse problem. They're so focused on creating great gaming content, they don't have time to make custom music or get right the rights to music that fits their brand and audience perfectly. Let's help both sides at once!
I formed and team and we started building. But we realized, why stop there? We might as well as help as many creatives as possible. We believe the best content is inspired with a clear vision and brand but produced with a diverse team of skilled creators who make sure every aspect is of the best quality. We want to expand this beyond music to DJs, pro gamers, augmented reality developers, video editors, or any other creative ideas people and bring to life on Twitch. That being said we're starting small and focused on streamers and music artists that to prove we can make something that both parties find value in, and eventually monetize.
What it does
We use custom machine learning algorithms that curate music for streamers based on what their audience for that specific game. We also have built a ducking and mixing app that allows streamers to turn down the music on their end as well as ducking it when they speak and their game makes loud sounds so they and their audience can focus on the critical elements of their stream and let the music fill in when things settle down.
Viewers initially will see (image 1) (after giving us permission to store their Twitch ID, so when they log in on another device or check out another stream we can still tie their Twitch and Spotify accounts togethr). (I'm using my personal Spotify account and songs for this and the following examples for clarity but will appear unique to each viewer. After clicking on the Spotify logo and signing in with a popup box, they will see (image 2). From here, they can choose to _ FOLLOW _ (the artist/s), _ add to playlist _, _ save to library _ (the latter two referring to the song). Clicking _ FOLLOW _ and _ save to library _ will immediately let the user know those actions have been done with the screen remaining the same. Clicking on _ add to playlist _ will have the panel display (image 3). We can see the user can scroll through their playlists and click the playlists they wish to add it to before hitting finish or simply scrolling back up to the stream. Should the viewer want to create a new playlist, they simply just hit that button and the panel will display (image 4).
Streamers play their music out of last.fm account where we will be automatically curating playlists for the streamers right in their last.fm accounts based on a stream scheduler in the config page (image 5) (UI needs some minor rebranding). The playlists will be made up of royalty-free songs as well as exclusively signed tracks from the Artist Intelligence Agency and Indie Music Group. We will be working on securing the rights from other labels and indie artists to grow the catalog even larger from our currently 5K tracks. Though we only have it working with last.fm as of now, we will be adding Soundcloud next, with the goal adding Apple Music and Spotify in the future for streamers to use. However, what is really exciting about what we've built is that we've designed custom machine learning algorithms which will learn what type of music a streamer's audience likes and then create and order playlists to optimize viewer retention. This means streamers grow their audience faster and Twitch keeps viewers on the platform for longer! This will take time and we will improve our algorithms as we get more data, so don't expect noticeable differences in the first few weeks (they have to learn)!
How I built it
This obviously was a team effort. However most of the devs I hired freelancers, and in the process of building out a more full-time in house team. I saved the money from two years of academic research along with some minor investment from friends and family. I will admit, with my limited software dev skills, I focused more on logic, architecture, and UI/UX while the rest of the team did the actual development. I took quite a bit of legal research and some pivots to make sure we are doing things by the book, and within extension guidelines.
The basic process has labels send us their catalog in which we store their mp3's on S3. We then to reduce the dimensionality and generate critical features using TensorFlow on AWS. We then take those features to find favorite songs and how to order them using AWS SageMaker. The data that is used is to train and organize is held on S3 and DynamoDB. We collect that data using Lamda functions to interact with the Twitch and last.fm APIs.
Challenges I ran into
It was tough cycling through different developers until landing on the team we have now. Stuff would be built and then they would leave or get a full-time job, essentially forcing us to start from scratch. Another struggle was that all my coding was limited to music and Academic research so I didn't have a full understanding which results of how things worked and also meant I could only give pseudocode directions. When working with mostly inexperienced developers this resulted in many challenges for everyone. The fact that it was almost all done remotely only compounded that. (Diving into Python is my next goal if I don't stay too busy with management)
We also learned just 3 weeks out (and I take responsibility for not clarifying this sooner) from the end that we couldn't link out of the extension so we had to hustle to figure how to build out all our functionality within the extension. But I'm really proud of our team of what we've able to spin out.
What I learned
This brings me to learning. I learned a lot about how to manage a team of now 5 (not including me). I always new communication was important, but finding the ways and frequency each developer prefers (and needs) following up with is something I feel like I have gotten much better with (but I am always working on getting better at).
I've also learned how to better focus on what "critical" features mean for an MVP. A lot of the issues we ran into we're not so much technical but rather confusion by trying to build out too many features at once before doing thorough alpha testing. This deadline has forced us to get to done is better than perfect and we have gotten so much more done in the past week or so because of it. I think I have a much clearer understanding of how to organize and set deadlines going forward.
I've also come to realize how critical and valuable it is for someone to have intimate knowledge about the space in which they're developing. I would say that music and entertainment experience is what I know best. My years of experience and intuition was critical in understanding what data sets we should with and then designing algorithms for feature extraction and playlist ordering were significantly better than seasoned data scientists who had less. I also found this same idea true with devs who did and didn't have significant experience in art / creative content. Regardless, front-end, backend, or data, this criterion was a demarker in how much time I had to spend focusing on and double checking their work and will be one of my key criteria for hiring moving forward.
Accomplishments that I'm proud of
Honestly being less than a year out of college and leading a team to make something that people are excited about is quite enough, especially being able to pursue all my passions of gaming, music, and tech. I would also say I'm proud of how much I got done while spending less than $35K (we've also built a desktop application that easily allows streamers to lower the volume of the music for themselves while keeping it louder for the audience, and duck the music based on mic and game sounds. We also have a website that's almost done in which Artists and Record labels can submit their music and get analytics on how it's performing).
What's next for gethr
I already mentioned how we want to start adding in other music platforms for both streamers and viewers. We will be continuously working on improving the UI/UX and adding not just platforms but also features. For example, having the user play and sync their own last.fm, Spotify, Apple music etc account with the stream so the artists get more accurate play counts, and viewers have control of the music volume. Another feature on the backlog is to create a _ Greatest Hits _ playlist which is made up the most saved songs and artists of the week for each streamer. Users will be able to easily follow the playlist which will be published on various platforms. This not only helps build brand recognition and generate revenue for the artists but also helps streamer develop an even greater sense of community and bonding with their fans.
I've talked with investors to help expedite the process, and being in Texas, there aren't educated in the California tradition of
Once we get to the point that music.