s01-p01: Introduction to Attention & Attention Pooling | CS4CV

Course: Computer Science for Computer Vision Channel: @CS4CV (   / @cs4cv  ) Playlist: Section 1 – Attention Mechanisms & Transformers (   • CS4CV Section 01: Attention Mechanisms & T...  ) In the first video of Section 1 – Attention Mechanisms & Transformers of Computer Science for Computer Vision (CS4CV), we introduce the fundamental idea of attention and build an intuitive understanding of how attention mechanisms work. We begin with the biological motivation behind attention, distinguishing between involuntary, saliency-based attention and voluntary, task-dependent attention. These ideas are then mapped to the core components of an attention mechanism: values, keys, and queries. Next, we introduce attention pooling through a simple Nadaraya–Watson kernel regression example. We start from a basic mean estimator and then show how predictions can improve by assigning different weights to different samples based on their relevance to the query. We then examine non-parametric attention pooling, where attention weights are computed using a Gaussian kernel and normalized with softmax. The attention weights are visualized to show that closer query–key pairs receive larger weights. Finally, we introduce parametric attention pooling, where a learnable parameter controls the weighting function and allows the model to adapt the attention behavior during training. Topics covered: Motivation behind attention Saliency-based and task-dependent attention Query, Key, and Value Attention pooling Nadaraya–Watson kernel regression Non-parametric attention Gaussian kernel attention Attention weight visualization Parametric attention pooling #Attention #DeepLearning #ComputerVision #MachineLearning #Transformers #CS4CV