Inspiration
Like many Gen Z, we grew up playing online games with friends. Although some of our fondest childhood memories took place on online games, it is vital to know that the online world can be just as dangerous as the real world.
As a game developer, Sammy became aware of an alarming yet little-discussed issue plaguing online gaming: the hidden presence of explicit content, particularly on children's platforms.
*With Apex, our mission is to protect children from being exposed to harmful content online. *
What it does
Apex is a tool which helps moderation teams locate and ban users who are in violation of a platform's rules. The algorithm does this by analyzing the connections (e.g. mutual friends) of known offenders, and identifying any accounts with a significant number of connections to offenders.
The identified accounts are then flagged for manual review by the moderation team. We elected not to automate the banning process because we believe in the right of all users to due process by a human being.
How we built it
We chose to deploy our system on the popular game Roblox, since 42% of its playerbase is under the age of thirteen. It is a platform we are familiar with, since Sammy has developed games on it.
Our frontend is convenient and comfortable, designed with the needs and priorities of moderation teams in mind. It was built using Bootstrap and connected to our Python-based system using Flask.
Challenges we ran into
As with most software projects, a significant challenge we faced was getting all of the technologies we used to work harmoniously. This included ensuring that our web-based interface was updating alongside the algorithm in a timely manner.
Another significant challenge was to minimize the API calls that we were making, optimizing the speed and performance of the program without overloading the Roblox API.
Accomplishments that we're proud of
We were shocked to find that the program worked far more effectively than we had anticipated. The sheer number of offending accounts it unearthed was at once amazing and disturbing. It managed to identify offending groups and users that we had never heard of, at a rate far faster than any human team could do so.
The accuracy of the algorithm was also quite amazing to us. Khalil manually reviewed many of the accounts it flagged, and almost all of them had correctly been identified as offenders.
What we learned
While the two of us have been discussing this issue for months, neither of us had truly realized the scale of it until we deployed the algorithm. The networks of malicious activity on children's platforms are far larger and more elaborate than we, and most people, ever knew.
What's next for Apex?
In Engineering, scaling a system can be as much of a challenge as inventing the thing to begin with. It is one thing to invent a solar panel, but to build a solar power plant is something entirely different. Likewise, our system has proven itself highly successful on the scale of several thousand users. The next step is to build a continuously-running system that can effectively moderate platforms with hundreds of millions of users - keeping them safe for the younger generations.
Log in or sign up for Devpost to join the conversation.