Logits
Raw, unnormalised scores output by a neural network before applying softmax.
Raw, unnormalised scores output by a neural network before applying softmax. The alignment scores eₜᵢ are logits — they can be any real number. After softmax, they become proper probabilities (attention weights).