BloomBERT

What we were inspired by:

We were curious about the process of text embedding and its intricacies. Most of our machine learning experience involved image classification, so we were excited about the opportunity to tackle this new problem.

What does the project do:

The program we built attempts to decode a series of encoded textual headline segments. After training a neural network, the program will make a prediction about what text corresponds to an encoded segment.

How did you build the project

We built a neural network in the hopes that the program could learn how to decode the encoded text segments. We tackled the problem by using an encoding scheme that we knew how to encode and decode, which was BERT. Using the examples of text and encoded data provided, we ran each headline text through the BERT encoder. The original encoded text arrays were set to be the input data for the neural network, and the associated BERT encodedings for the same text were set as the labels. We trained a neural network in python using tensorflow to associate the original encodings to the BERT ones. When the network outputted predictions as BERT encodings, we then decoded the BERT encodings to obtain our final decoded text output.

What challenges did you face:

Our primary challenge was that we tried to implement a transformer instead of a sequential neural network, but couldn't manage to get past errors. In particular, we had issues trying to add a MultiHeadAttention layer into that model.

What are you proud of:

We were able to run training for the decoder, and even though the output predictions weren't consistent, we are proud of the fact that we managed to get it to output a textual prediction.

What did you learn:

We learned about text encoding and the process for generating it, as well as uses at Bloomberg. Although we didn't end up using one, we learned about transformers and their usefulness in learning text related tasks.

What's next for BloomBERT

In terms of what's next for BloomBERT, it would be very interesting to try to get a transformer model to train and see what potential changes in results and accuracy it would have.

Built With

• google-colab
• keras
• python
• tensorflow