Overview

Learning transferrable representations from large raw text has been one of the key reasons behind recent advances in natural language processing (NLP). Deeper representations are trained thanks to more powerful computing hardware and more parallelizable neural network architectures. As a general practice, for training a model for a target task, we moved from transferring only static pre-trained word embeddings (e.g., word2vec) to transferring deep layers of pre-trained neural networks interpreted as contextual word representations (e.g., BERT). Models like BERT represent meanings of words and phrases in either their embedding layer or the activations of their hidden layers or both. In this talk, after a brief background on pre-trained word representations, I will focus more on how different meanings are represented in word embeddings and introduce our method of probing semantic classes. We use semantic classes as proxies for meanings and train classifiers to tell us which meanings are encoded in word representations. We also apply a similar approach for contextual word representations like BERT.