Objective: understand Query, Key, Value, multi-head attention and Transformer blocks.
Attention lets every position weight the information coming from other
positions. From a representation X, we compute:
Q = XW_Q, K = XW_K, V = XW_V
Attention(Q,K,V) = softmax(QKᵀ / √d_k) VThe query Q describes what a position is looking for, the keys K what
each position offers, the values V the information to aggregate. Dividing
by √d_k avoids scores too large that would saturate softmax.
Several heads project the data into different subspaces, apply attention in parallel, concatenate their outputs, then reproject them. A head is not guaranteed to correspond to a stable human concept.
Preview — the rest of the lesson is for enrolled readers.
Your access is tied to your account, not to this link. Sign in with the same email you used in class: your course is waiting, no need to enter the code again.
Sign inNo account yet? Create oneThe first modules of the course are open to everyone. For the rest you have three options: buy this course once and for all, subscribe, or enter the code handed out in class.