Quinfinit

Motivation

In language assessment, cloze (fill-in-the-blank) multiple-choice questions are commonly used to test students' knowledge of vocabulary. However, they are labor-intensive to create, and it can take over an hour of an educator's time to create a quiz of 20 questions with distractors. We propose to generate cloze questions automatically using GPT-3.

Solution

Our approach consists of two steps: question generation and distractor generation. To generate the question, we prompt GPT-3 to write a sentence using a word using the completion API with the few-shot learning setting.

Good distractors should be semantically similar to the keyword, be the same part of speech, but not too close to the keyword that it's a valid answer. To generate distractors, we take the nearest neighbors of the keyword from GLoVE, filter out the words that mismatch on part-of-speech or morphological features, and re-rank the remaining ones using GPT-2 (as the GPT-3 API does not allow calculating perplexity). We exclude the words with lowest perplexity (indicating high cloze acceptability), so the remaining ones are valid distractors.

Challenges

One difficulty was the lack of control over GPT-3 outputs: there was no way to guarantee that the generated sentence actually contained the keyword. We resolved this difficulty by generating multiple completions from GPT-3, increasing the likelihood that at least one of them would contain the keyword.

Built With

• flask
• gensim
• gpt-3
• huggingface
• pytorch
• react
• spacy