Attention

Humans are able to naturally determine the contect behind a word which can have multiple meanings (homonym) for example, differentiating between when "spring" means a season or a metal coil. However, for large language models, this process is known as "attention". Attention mechanisms are therefore integral to large language models, allowing for their ability to understand and generate natural language.

In machine learning, there are two types of attention typically talked about:

     1) Self attention: a core mechanism in models like transformers, it evaluates the importance of elements within the same input sequence (e.g., words in a sentence) by computing attention scores among them. This enables the model to assign varying degrees of importance to each element based on its relationships with others, leading to richer context understanding.

     2) Multi head attention: extends self-attention by employing multiple parallel attention mechanisms, or "heads." Each head focuses on different aspects of the input, allowing the model to capture diverse relationships and representations simultaneously. These individual heads are then combined to provide a comprehensive understanding of the data, enhancing model performance in various machine learning tasks.