Objective: turn discrete symbols into learnable representations.
An embedding maps each discrete identifier to a dense vector. A
vocabulary table of size V and dimension d contains V×d parameters.
from torch import nn
embedding = nn.Embedding(num_embeddings=30_000, embedding_dim=256,
padding_idx=0)Two nearby vectors sometimes have similar uses, but the geometry depends on the task and the corpus. It can also encode biases present in the data.
Tokenization splits text into units and converts them to identifiers. Subword methods limit unknown words by combining fragments. The tokenizer, its vocabulary, normalization and special tokens are an integral part of the model.
Preview — the rest of the lesson is for enrolled readers.
Your access is tied to your account, not to this link. Sign in with the same email you used in class: your course is waiting, no need to enter the code again.
Sign inNo account yet? Create oneThe first modules of the course are open to everyone. For the rest you have three options: buy this course once and for all, subscribe, or enter the code handed out in class.