Project Checkin 2
Introduction:
This project is to examine the effectiveness of SOTA meta-learning algorithms on top of pre-trained language model (i.e. BERT) on natural language understanding (NLU) tasks (i.e. GLUE dataset) and further on multimodal tasks (i.e. question answering, image captioning)
Challenges:
The biggest challenge so far for us has been resource management and debugging performance issues. Our program is exhausting huge amounts of memory and disc space. Because we are using a pretrained model, we don't have great insight into what is going on under the hood and how it can be managed better.
Insights:
Unfortunately we are yet to run our program on the full data set. While our code runs and produces results on a subset of the data, as mentioned above it crashes on the full data set. There are no viable point of references for the expected performance on a subset of data so we are not able to make any assessments regarding the performance of the model at this time.
Plan:
The absolute priority is to fix the performance issues since the next step is to run tons of experiments and we can not do it without a fully functioning code. The original plan was to test and compare 4 different implementations (MAML, REPTILE, LEOPARD, SMLMT), in light of the current setbacks we have decided to drop LEOPARD and SMLMT and focus on the former two