A large language model does not retrieve a stored answer. It reads your text as small pieces called tokens, scores every possible next piece, writes one, and repeats. That is why llama3.2 can write a fluent sentence about NorthPeak even though the company does not exist in its training data. This lesson names the parts of that machine: tokens, next-token prediction, attention, parameters and the context window.
A model does not read letters or words. It reads tokens. A token is a small piece of text. Short common words are one token. Long or rare words are cut into several tokens. Montreal may be one token. vibration_mm_s is cut into four or five.
The model has one job: guess the next token. You give it "The pump in Toronto is". It computes a score for every possible next token. "running" gets a high score. "purple" gets a low score. It picks one. Then it adds the token to the text and starts again. One token at a time, it writes a whole answer.
This is called next-token prediction. It looks simple. But to guess well, the model must have learned grammar, facts and reasoning patterns. It learned them by reading a very large part of the internet.
The architecture that made this work is the Transformer, from 2017. Its key idea is attention. For each token, the model looks at all the other tokens in the text and decides which ones matter. This is why it can link "it" to "the pump" ten words earlier. You will meet attention again in Week 8.
A model is made of numbers called parameters. Training sets their values. More parameters means more room for facts and patterns. That is why bigger models know more. But bigger models need more memory and run slower.
The model of this course is llama3.2. Run ollama show llama3.2 in your terminal. Part of the output:
Model
architecture llama
parameters 3.2B
context length 131072
quantization Q4_K_MThree lines matter. parameters 3.2B: 3.2 billion numbers, a small model. It takes 2 GB on disk and runs on a laptop. context length 131072: the model can read up to 131,072 tokens in one call. quantization Q4_K_M: each parameter is stored in about 4 bits to save memory.
The context window is the maximum number of tokens the model can read in one call. Your prompt, the earlier messages and the answer all fit in it. Anything outside the window does not exist for the model. Ollama sets a smaller default window, 4,096 tokens, to save memory. You can raise it if you need to.
In Exercise 2 you will ask llama3.2 about NorthPeak. NorthPeak is a fictional company. Its data was created for this course. The model never saw it. So it has nothing to predict from. It will either say it does not know, or write something that sounds right. Both come from the same next-token machine.
The model predicts what text is likely. It does not check what is true. A likely sentence and a true sentence are often the same. Not always. When the model has no data, likely wins over true. That is a hallucination. The fix is not a bigger model. The fix is to give the model the data. You will do that in Exercise 3.