LABLS.IO - Human Computation Experiment for Image Labelling

Recent figures estimate humans spend billions of hours playing games annually. What if we could capture all this creativity and power to solve an outstanding computational issue of image labeling?

Comment 1

Inspiration

A few months ago the 'Ten Year Challenge' took social media by a storm. Users on platforms such as Instagram and Facebook would upload a side-by-side comparison of themselves a decade ago and now. Thinking of the amount of data this could have provided for a Machine Learning ' ageing' algorithm, we questioned if we could also harness the power of social media and enjoyment in order to help solve large-scale computational problems.

This hack was inspired by Games with a Purpose by Luis von Ahn IEEE Explore https://www.cs.cmu.edu/~biglou/ieee-gwap.pdf

Problem we're trying to solve

Image labelling is a large requirement for several machine learning applications. This is quite a difficult process to automate as it requires human judgement to decide what the main focus of an image and the overarching theme of objects present within is. Often time this is out-sourced to manual clicking which is tedious and laborious. Creating labls not only aims to provide a more incentive-driven aspect to image labelling through enjoyment, but also improves on this and harnesses the ability of humans to naturally create links and categories between images themselves.

What it does

labls is a 4 player game inspired by the classic board-game 'Codenames.' It initialises a 2x4 grid of images for all players to see on a web page, which are then sorted between the teams, 4 belong to the red team, 3 belong to the blue team and 1 card is the designated killer card. A captain is then selected on each team and is informed of the card allocations. Each captain then tries to find a word which relates as many of their photos together. This word is then given to their teammate along with a number representing how many of the photos are described by this word. For example, if a team had designated images of: 'dog', 'cat', 'rocket' and 'car', might result in a captain passing: "Vehicle, 2" or "Animal, 2" along to their teammate. Every correctly chosen image by the teammate gives the team a point. It is in the best interest of the captain to avoid choosing a word that may cause their teammate to select one of the opposition's images as this gives them a point. More importantly, if the killer card is chosen the result is an instant loss. So a killer card of 'hedgehog' would make the clue: "Animal, 2" risky as it would prompt the teammate to select the killer 'hedgehog' card. This is advantageous to the overall project as it will result in more detailed labelling and categorisation.

How we built it

First we discussed what the structure of the software should be and how the front-end would communicate with the back-end. The back-end modelled a server and was written in Python while the front-end code was done in JavaScript. The web design was implemented in HTML.

Challenges we ran into

The main difficulty was in linking up the front-end with the back-end. We had to change the structure of the Python program as the while loops we had used originally caused the front-end code to wait indefinitely.

Accomplishments that we're proud of and what we learned.

Coming into the competition as four freshers and an engineer, we were all eager to build our first project from scratch; it was exciting to be able to bring our idea to life in such a buzzing environment and the materialisation of the project showed us the potential to make other projects collaboratively in the future, as well as having a strong platform for us to carry on working on after the Hack! It was exciting to be able to learn two programming languages as well as being introduced to full stack development for the first time.

What's next for LABLS.IO

To improve image categorisation we could add bonus points which would involve the captain picking which picture he thinks has the highest chance of being chosen by his teammate. This is a direct label from a word to a picture. Similarly, if our client requires their images to be classed by adjectives we could implement bonus points for adjectives. We could also implement ways to vary the team size and display photos in different combinations to categorise the photos against each other. Lastly, from a mathematical point of view the game could be used to construct models and graphs of links between images.

Value proposition

There is a clear motivation for all parties involved. A client who may want their photos categorised can send in the photos they wish to be labelled. As described above, future versions of labls will be very flexible and would allow the client to customise their labelling. The people playing the game will have fun, often seen as desirable.

Share this project:

Updates