With BitWise, we can significantly increase the number of people who can help label datasets and do it at a greater speed than ever before.
Datasets for machine learning like ImageNet have created so much value and are an incredibly important part of the future. However, barely anyone can help label them. With over a thousand classes, even excluding the prevalence of very specific nouns and species of animals like birds, the task of knowing every class in such a large dataset is incredibly hard.
We wanted to improve this and allow anyone to help researchers and companies label datasets while making a little money in the process.
What it does
Instead of requiring people to learn the intricacies of image datasets, we propose treating image labeling like a game of 20 (or N) questions. Users are tasked with answering the same binary question for groups of images of the same decision node. As users answer these questions, images are propagated further down the tree iteratively until they are classified. Users gain points for each image they classify and can redeem these in the future for gift cards.
There are multiple advantages for this solution. By giving users much of the same mental task in a row, people don’t have to constantly shift their attention to different classification tasks and can therefore work more efficiently and have greater accuracy.
One challenge arises in this solution: how do you produce binary questions at scale for a very large, specific dataset? For this, we use WordNet, a NLP dataset that encodes the hyponyms and hypernyms (supersets and subsets of words) in a tree. Since any tree can be binarized, we can create a new question set for almost any dataset instantly. We can use a Least Common Ancestor algorithm to find the lowest common branch of the tree for all needed categories and shake the tree to decrease the number of classifications needed per image.
With BitWise, we can significantly increase the number of people who can help label datasets and do it at a greater speed than ever before. People can also make money in an ethical and equitable way in the process.
How I built it
Our backend is built with a REST API in Flask connected to a MongoDB database in Python. One of the advantages of using a non-relational database like MongoDB is that we can represent our database like an actual tree. We calculate the tree structure using WordNet, a dataset that maps the semantic field relationships (supersets and subsets of words) between words in a tree-like structure, binarizing the tree. We implemented a custom scheduler to navigate the tree and find the next best images a user should classify. We also host our static images on-server. Currently, our app is configured to use a subset of ImageNet called TinyImageNet. However, this is easily configurable. This is all deployed on Google Cloud.
Challenges I ran into
From a project management perspective, one interesting problem was developing common interfaces for multiple moving components of our project, especially front-end and backend.
What I learned
We learned a lot about APIs, non-relational databases, and software engineering best practices.
What's next for BitWise
We hope to integrate fault-tolerance into the product, as well as develop a fraud detection model. The fraud detection module would randomly ask the user to answer a binary classification problem for a prelabeled image as a test of user aptitude. In addition, it would remember players who consistently fail the aptitude test and/or give answers that turn out to be outliers. These players would receive a punishment that leads to their answers having less weight compared to other users. Likewise, players who consistently pass the aptitude test and give strong answers will receive a reward that increases their importance. Players could receive some indication of their importance and perhaps receive a monetary reward if they do particularly well.
We could also attempt to gamify the experience by adding a player leaderboard, with players ranked by number of questions answered. Another idea is to add daily quests that lead to a monetary reward. Both of these ideas make the users feel more connected with one another. Furthermore, they provide a competitive push for them to classify more images.