Of Mice and Men

Link to Final Writeup

https://docs.google.com/document/d/1h_bm6t56jQw-01kE-H16MxxtlhM_6LFUkEuTIt-bUPI/edit?usp=sharing

Everything below is from our first delieverable. Please refer to the final write up for most updated details about the project.

Introduction

We are planning to implement Mendelsohn et al. 's computational linguistics model that provides a framework to analyze dehumanization in text data. The main objectives of this paper are to computationally operationalize prior research on dehumanization, such as research that shows that comparisons of social groups to vermin (like rats or parasites) are used to dehumanize that group. There are four main topics that the authors seek to computationalize: negative valence towards the target group, denial of agency to the target group, association of moral disgust with the target group, and comparison of the target group to vermin. Using this framework, the authors were able to track how the term “gay” has become less dehumanizing over time, but the term “homosexual” is still quite dehumanizing. The basic premise of this project is that the model trains word embeddings through a prediction task (predicting the next word in a sequence), and then the model evaluates dehumanization through cosine similarity comparisons to vectors (such as a vector representing moral disgust). Our group was interested in this project because we wanted to do something related to linguistics as well as something that could be used for social good.

Related Work

One of the foundational papers that inspired the current paper that we’re trying to implement is “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings” (Bolukbasi 2016). This paper demonstrates that you can evaluate gender bias of news articles simply by training word embeddings on a dataset of news articles, and then using cosine similarity to evaluate the relationship between words (such as “woman” and “homemaker”). This paper shows that you can evaluate the connotation of a word by simply looking at its nearest neighbors in vector-embedding space, which is a concept used in the paper that we’re trying to implement as well.

We aren’t aware of any public implementations of the complete paper.

Data

Our data contains New York Time articles from January 1986 to December 2015. We contacted the authors of the paper, and they provided us with their original data in a tsv file and gave us permission to use this dataset. The tsv file contains each row as a paragraph from an article, so each article is divided into its paragraphs. The file also contains information about the article’s published time and category, as well as a unique id for each article.

The entire tsv file is a little over 13 GB. The articles in the tsv file we have are divided into paragraphs, so we will likely need to aggregate all paragraphs for each article before preprocessing further. This process shouldn’t be too tricky since each article has a unique ID in the original tsv file. Beyond that, the file doesn’t seem to have tokenized sentences, so we will need to work on cleaning the article paragraphs a bit, such as tokenizing, creating a vocabulary, UNKing the data, and converting words into vector representations. After that, the model can utilize the cleaned data.

Methodology

The foundation of the paper is a word2vec embedding based on the input text, so our first step would be training a simple word2vec model to create word embeddings. (The original paper did not seem to suggest that they used any off-the-shelf word embeddings, eg. GloVe.) After we have the model trained, we will take a group of words and analyze how close (based on cosine similarity) their word vectors are to word vectors with dehumanizing denotation. Based on the paper, dehumanizing factors can include: words that reflect denial of agency, words that reflect moral disgust, words that relate to vermin as a dehumanizing metaphor. We think that analyzing these three different dehumanizing factors will not be particularly hard, since the main task is training word embeddings.

Metrics

The base goal of the project is to build our own word2vec model, and the target goal would then be re-implementing the paper using the word2vec model and doing the analysis on terms as mentioned in the paper.

One straightforward way of measuring our model is comparing our results with the paper’s, assuming the paper’s result reflects a relatively accurate representation of the topics in focus. For example, we can plot cosine similarities between different vectors and compare the nearest neighbors for certain word vectors to the ones presented in the plots in the paper.

We are still figuring out how to define accuracy in other ways. The main evaluation from the paper was based on human annotations where they gather opinions from Mechanical Turk workers on how dehumanizing a sentence is, and comparing their word embedding analysis against the human judgement. One direct way of evaluating our model would be doing a reduced version of human evaluation, i.e. sampling a group of sentences from the model’s prediction and determining if those sentences contain dehumanizing factors ourselves.

A possible stretch goal could be using our model on different terms, so that we can measure dehumanization of other social groups, or training our model on different corporas (such as presidential speeches) and see if we notice any different trends in use of dehumanizing language. A more difficult stretch goal could be the incorporation of the current model into a larger off-the-shelf sentiment analysis model, if time allows.

Ethical considerations

This problem is clearly relevant to societal issues regarding LGBTQ discrimination and how language contributes to and reflects specific ways that LGBTQ groups are dehumanized. The model implemented in this project helps quantify how much discrimination of LGBTQ groups appears in writing, confirming that such discrimination associates LGBTQ people with sentiments of disgust and low agency. Thus, this problem also investigates how unfair semantic associations are embedded in our usage of language (perhaps even unknowingly) on a general level. This means that the model can easily generalize to other forms of discrimination: investigating the valence of words referring to certain racial, religious, ethnic, or socioeconomic groups.

One major stakeholder in this problem is the New York Times and, by default, the journalism industry at large. While the dehumanizing associations discovered in this paper are certainly not limited to journalism, it quantifiably shows that such publications have a history of using LGBTQ terms in negative ways. Mistakes in our algorithm could display the New York Times and its writers as more or less discriminatory than they actually are. Other stakeholders are readers, who could possibly behave differently or seek out different media depending on the results of this paper. Lastly, a major stakeholder is the LGBTQ community, whose lives are affected everyday by the language that this model investigates. Mistakes in our algorithm could improperly deny their marginalization and ignore problems for which real solutions must be found.

Division of labor

Andy and Ethan will work on preprocessing the data. Holly and Becky will work on creating and training word embeddings and setting up the training architecture. Once we have a basic model architecture in place, we’ll probably regroup and discuss division of labor again as it pertains to creating comparison vectors (denial of agency, moral disgust, vermin as dehumanizing), analyzing metrics, and later on start working on stretch goals.

Built With

• word2vec