Overview

In both text and speech processing, variants of the Transformer architecture have become ubiquitous. The key advantage of this neural network topology lies in the modeling of pairwise relations between elements of the input (tokens): the representation of a token at a particular Transformer layer is a function of the weighted sum of the transformed representations of all the tokens in the previous layer. This feature of Transformers is known as ‘context mixing’ and understanding how it functions in specific model layers is crucial for tracing the overall information flow. In this talk, upon reviewing prior efforts to quantify context mixing, I will introduce Value Zeroing, and show that the token importance scores obtained through Value Zeroing offer better interpretations compared to previous analysis methods in terms of plausibility, faithfulness, and agreement with probing. Next, by applying Value Zeroing to models of spoken language, we will see how measures of context mixing can reveal striking differences between the behavior of encoder-only and encoder-decoder speech Transformers.