Intuition of Attention
Our brains have the ability to attend to important parts of an image or information effortlessly. For example, when we look at an image of Iron Man, we can immediately identify the important parts and focus on them. This process involves our brain identifying which parts to attend to and extracting the features that deserve the highest attention.
This concept is similar to the idea of search. When we search for information on the internet, we enter a query and the search engine finds relevant results by comparing the query to the key information in its database. The search engine computes a metric of similarity and relevance between the query and the key information. This similarity computation is crucial in determining the important parts of the search results.
To understand how self-attention works in neural networks like Transformers, let’s consider a language example. Our goal is to identify and attend to features in a sentence that are relevant to its semantic meaning. Instead of processing the sentence sequentially, we encode and capture the order information using positional encoding. This encoding captures the relative relationships and allows us to process all the time steps at once.
Next, we use neural network layers to generate the query, key, and value from the…
