BERT Sentiment Analysis

• 

  Demo of app

Inspiration

I have been involved in NLP research for about 6 months now, and I am interested in learning more about state of the art models such as the BERT family of transformers.

What it does

This application fine-tunes a pretrained DistilBERT model on a subset of the GLUE:sst2 dataset over a variable number of epochs, and performs sentiment analysis on user input.

How we built it

I made heavy use of the Huggingface and Torch libraries, and their documentation.

Challenges we ran into

The F-1 score achieved by my model when evaluated with a held out test set tends to be low at 0.6 - 0.7. I believe this is because I do not provided enough training data over enough epochs. Doing so would take a really long time.

Accomplishments that we're proud of

The application actually works pretty well. The accuracy score hovers between 0.8 - 0.85 when the model is evaluated with a held out test set, which translates into fairly accurate sentiment analysis for user input.

What we learned

I learned a lot about the Huggingface API, the tqdm library, and the Torch library.

What's next for BERT Sentiment Analysis

I want to incorporate principles of active learning as they can be applied to fine tuning pre-trained models for downstream tasks, using larger subsets of data, and increasing the performance of the model as measured with a held-out test set.

Built With

• bert
• elsa
• huggingface
• python
• torch