Chapters 001 to 003 placed supervised learning inside the taxonomy of the field and separated the three learning paradigms. This chapter moves from placement to mechanics: what a supervised problem contains, what the algorithm does while it fits a model, and what is actually being measured when the model is evaluated.
The definition is given at four levels of decreasing precision. All four state the same thing. The first must be mastered; the other three exist so the idea can be transmitted to different audiences without becoming false.
The spaces. An input space X (typically X ⊆ R^d, where d is the number
of explanatory variables) and an output space Y. The nature of Y fixes the
type of problem: Y ⊆ R gives regression, and Y = {c1, ..., cK} finite and
unordered gives classification.
The generating distribution. Observation-label pairs are assumed to be drawn
from a joint distribution P(X, Y) that is unknown and assumed stable
over the period considered. Drop that assumption and nothing observed says
anything about what has not been observed. The whole discipline rests on it.
The dependence assumption. A relationship between X and Y is postulated
and written as an unknown function f : X → Y, or — in the general case where
the relationship is noisy — as the conditional law P(Y | X). The standard model
is
y = f(x) + εwhere ε is a random term with zero expectation standing for everything that
moves y without being captured by x.
The sample. Neither f nor P is available. What is available is a finite
sample
D = {(x1, y1), (x2, y2), ..., (xn, yn)}, (xi, yi) ~ P(X, Y)of n observations assumed independent and identically distributed.
The criterion. A loss function L : Y × Y → R+ quantifies the gap between a
predicted value and an observed one. The expected risk of a hypothesis h is
R(h) = E[L(y, h(x))], the expectation taken over (x, y) ~ P(X, Y)The theoretical objective is h* = argmin R(h) over the hypothesis space H,
the set of functions that the chosen model class can represent.
The obstacle, and the way around it. R(h) cannot be computed, because P
is unknown. It is replaced by the empirical risk
R̂(h) = (1/n) · Σ L(yi, h(xi)) for i = 1..nand the tractable problem f̂ = argmin R̂(h) over H is solved instead. This
substitution is the principle of empirical risk minimization (ERM), and it
is the formal foundation of supervised learning. Everything that follows in this
course — data splitting, regularization, cross-validation, leakage hunting — is
a consequence of the gap between the quantity that matters and the quantity that
can be computed.
Rigorous definition
The expected risk is the expectation of the loss under the generating
distribution: R(h) = E[L(y, h(x))]. It measures the performance of h over
every observation the distribution can produce, observed or not.
The empirical risk is the mean loss over the available sample. It is an
unbiased estimator of R(h) for a hypothesis fixed in advance.
Empirical risk minimization (Vapnik, 1995) consists in retaining the
hypothesis that minimizes R̂ over H, as a substitute for the inaccessible
minimization of R.
Central point of caution
Unbiasedness holds for a hypothesis fixed before the sample is seen. As soon as
f̂ is selected by minimizing R̂, the quantity R̂(f̂) becomes an
optimistically biased estimator of R(f̂). Selection consumes the
independence that made the estimator honest. This is the formal justification for
splitting the data (chapter 026) and the mechanical origin of overfitting
(chapter 031).
In plain terms
You want the model that is wrong least often across every case that could occur; you can only measure error on the cases you happen to hold. So you pick the model that is wrong least often on those, knowing the resulting score flatters it.
The gap between R(f̂) and the best achievable performance decomposes into four
distinct error terms. Knowing which term dominates determines which action is
worth taking; acting on the wrong term wastes effort.
| Error term | Origin | Lever for reduction |
|---|---|---|
| Approximation | The best function in H is still far from f | Enlarge H: a more expressive model, additional variables |
| Estimation | n is finite, so R̂ estimates R only imperfectly | Increase n, restrict H, regularize |
| Optimization | The algorithm does not reach the exact minimum of R̂ | Tune the optimization procedure |
| Irreducible | The noise ε: y is not determined by x | None. It bounds achievable performance |
Enlarging H lowers approximation error and raises estimation error. That
trade-off is the bias-variance decomposition, treated in chapter 032. The fourth
term deserves particular attention: irreducible error is not a defect to be
engineered away. A project whose performance objective sits below the level the
irreducible error permits is not ambitious; it is ill-posed.
Supervised learning fits a parameterized function on a set of examples whose value to predict is known, by minimizing a measure of discrepancy between predicted and observed values, so that the function can afterwards be applied to observations whose value is unknown.
This is the statement for a design document. It names the four mandatory elements: labeled examples, a parameterized function, a discrepancy measure, and application to the unknown. Remove any one of the four and the sentence stops describing supervised learning.
An algorithm is given a large number of past cases together with their answer. It searches for the rule that best connects the available information to the answer. That rule is then applied to new cases, for which the answer is not known.
This is the statement for a steering committee. It contains no false approximation; it simply omits the formalism.
How to read it : the arrow Y → ALG exists only during training. That arrow,
and nothing else, is what separates supervised learning from unsupervised
learning (chapter 003).
| Level | Statement | Audience | Where it is used |
|---|---|---|---|
| 1 | Mathematical: f̂ = argmin R̂(h) over H, D ~ P(X,Y) | Data scientist, examiner | Technical interview, specification |
| 2 | Technical: a parameterized function fitted by minimizing a discrepancy | Engineer, architect | Design document |
| 3 | Everyday: past cases with answers, an induced rule, applied to the new | Executives, business owners | Steering committee |
| 4 | Graphical: two phases, one discriminating arrow | Any audience | Presentation material |
| Symbol | Meaning |
|---|---|
x, y | Vector of explanatory variables, observed label |
n, d | Number of observations, number of variables |
P(X, Y) | Joint generating distribution, unknown |
D | Training sample, n pairs |
f, f̂ | Unknown true relationship, retained fitted model |
ŷ | Predicted value, ŷ = f̂(x) |
H, L | Hypothesis space, loss function |
R(h), R̂(h) | Expected risk (not computable), empirical risk |
Rigorous definition
Supervision denotes the availability, for every observation in the training
sample, of a reference signal yi giving the value the model must learn to
produce. That signal plays the role of a supervisor in the control-theory sense:
for each attempt it supplies the error information that makes correction
possible.
Origin of the term
The phrase supervised learning comes from the neural network literature of the 1950s and 1960s, which distinguished procedures that had a teacher signal from those that did not. The supervisor is the signal, not a person.
Point of caution
Supervision is a property of the training data, not of the operating regime of the system. A deployed supervised model can run with no human involvement at all; conversely, an unsupervised system can sit under permanent human control. The two questions are independent, and confusing them leads teams to assume oversight that was never built.
| What "supervised" does mean | What "supervised" does not mean |
|---|---|
| A reference label for every training observation | A human operator present while the system runs |
| An error signal computable at every attempt | Quality control over predictions in production |
| A property of the learning phase | A property of the deployed system |
A student prepares for an exam using a workbook in which every exercise comes with its solution. For each exercise the student proposes an answer, compares it to the key, sees the gap, and adjusts the method.
| Element of the analogy | Formal element |
|---|---|
| The exercise statement | The vector x |
| The solution in the key | The label y |
| The proposed answer | The prediction ŷ |
| The observed gap | The loss L(y, ŷ) |
| Adjusting the method | Updating the parameters |
| The final exam on unseen problems | Evaluation on the test set |
The answer key is not a teacher watching the student during the exam. It is reference information available during preparation and unavailable on exam day. That is exactly the status of a label.
The analogy also carries the characteristic failure. A student who memorized the answer keys without acquiring the method scores perfectly on the workbook and fails the exam. That case is overfitting, and the analogy predicts its signature: excellent on what was studied, poor on what was not.
Rigorous definition
A labeled datum is a pair (x, y) associating a vector of explanatory variables
x ∈ X with the value y ∈ Y of the target variable for that same observation,
where that value has been observed, measured, or assigned by a process
external to the model.
In plain terms
A row in a table that carries, alongside the descriptive information, the answer you want to be able to predict.
Existence condition for supervised learning
No labels, no supervised learning — without exception. The alternatives are then: create the labels (annotation, instrumentation, waiting for history to accumulate), reformulate the problem as unsupervised (chapter 003), or abandon it. There is no fourth option, and no algorithm choice that rescues the first three.
Does the available data contain a column holding exactly what you want to predict, populated for past observations?
| Answer | Consequence |
|---|---|
| Yes, populated and reliable | The problem is supervised. The rest of this course applies |
| Yes, but partially populated or of doubtful quality | Supervised with reservations. Label quality becomes the first workstream |
| No | The problem is not supervised as it stands. Either create the label or change paradigm |
Point of caution : "exactly what you want to predict" is a strict
requirement. Predicting customer satisfaction from a column named
reopened_a_ticket predicts ticket reopening, not satisfaction. The two may
correlate; they are not the same variable, and the model will optimize the one
that is in the column.
Example A — Spam detection
| known_sender | link_count | uppercase_pct | has_attachment | is_spam |
|---|---|---|---|---|
| Yes | 1 | 4 | No | No |
| No | 14 | 62 | Yes | Yes |
| Yes | 0 | 2 | Yes | No |
| No | 9 | 48 | No | Yes |
The first four columns are x; the bold column is y, valued in {Yes, No}:
binary classification. The label comes from the user's own action, through a
manual report.
Example B — Predictive maintenance on instrumented equipment
| vibration_mm_s | temperature_c | hours_since_service | cycles_per_day | failure_within_7d |
|---|---|---|---|---|
| 2.1 | 64 | 820 | 142 | No |
| 5.8 | 81 | 2,340 | 168 | Yes |
| 3.4 | 71 | 1,510 | 151 | No |
| 6.2 | 88 | 2,780 | 174 | Yes |
y is binary: classification. The label is reconstructed after the fact
from the maintenance log — for each reading, check whether a failure occurred
within the following seven days. That reconstruction is the most delicate
technical step in the whole case, and it is developed in section 7.
Example C — House price estimation
| area_sqm | bedrooms | year_built | distance_to_center_km | sale_price |
|---|---|---|---|---|
| 75 | 2 | 1994 | 6.2 | $272,500 |
| 100 | 3 | 2006 | 4.1 | $370,000 |
| 125 | 4 | 2011 | 3.4 | $467,500 |
| 150 | 5 | 2018 | 2.8 | $560,000 |
y is continuous and numeric: regression. The label is a recorded fact, the
price written into the deed of sale.
The three examples carry labels of radically different natures: a user report, a reconstruction from a log, a contractual amount. Those differences determine the reliability of the resulting model far more than the choice of algorithm does. An excellent algorithm on poor labels produces a poor model; a mediocre algorithm on excellent labels produces a usable one.
| Source | Description | Cost | Reliability | Associated pitfall |
|---|---|---|---|---|
| Natural history | The event occurred and was recorded: sale closed, contract canceled, claim filed | None to low | High | The system's definition of the event may differ from the business definition; the recording date and the occurrence date are not the same date |
| User action | The user produces the signal: click, purchase, report, rating | None | Medium | Massive selection bias — only a minority acts. Absence of a signal is not a negative signal |
| Human annotation | Operators label observations specifically for the project | High | Variable | Subjectivity, fatigue, criterion drift. Requires an inter-annotator agreement measure |
| Domain expert | A specialist qualifies each case using professional judgment | Very high | High on typical cases | Low throughput, therefore low volume. Experts disagree on borderline cases, which are precisely the cases of interest |
| Automatic business rule | An existing rule produced the label: threshold, rules engine, expert system | None | Illusory | The model learns the rule, not the phenomenon. Its performance is capped at that of the system that produced the labels |
| Instrument measurement | A sensor supplies the value: probe, meter, analyzer | Medium | High | Calibration drift, failures that emit default values, temporal desynchronization |
The question to settle at the start of any project :
How was this label produced, and is that production process reliable?
It comes before the choice of algorithm, before the choice of variables, and before any architectural consideration. It is settled with the people who operate the source system, not with the people who operate the database. The database team can tell you what the column contains; only the operations team can tell you what has to be true for the column to be filled in.
Situation
An organization has run a rules engine for years to classify incoming requests automatically. The history therefore contains, for every request, the class the engine assigned: a column immediately available, apparently an ideal label.
What happens
The model is trained to reproduce the engine's decisions. It succeeds, often above 95% accuracy. That accuracy measures fidelity of imitation, not correctness of decision. Every error the engine made is learned as truth: in the data, those errors are the definition of the right answer.
A supervised model cannot be better than the process that produced its labels. It can be faster, more consistent, cheaper to run, and it can generalize to cases the explicit rules never covered. It cannot be more correct.
Practical consequence
Such a project remains legitimate when the goal is to cut execution cost, cover cases the rules do not handle, or replace an engine that has become unmaintainable. It is illegitimate when the stated goal is to improve decision quality. In that case a label source independent of the system being replaced is required — an audit of a sample, or observation of the real-world outcome of the decisions.
A supervised learning algorithm performs no analysis of the domain. It runs a three-step cycle, repeated until a stopping criterion is met.
How the correction is carried out varies: gradient descent for parametric models and neural networks, recursive partitioning by maximizing an information gain for trees, a closed-form analytical solution for ordinary least squares. The logical structure of the cycle is identical in all three.
Rigorous definition
A parameter is a quantity internal to the model whose value is determined by the learning algorithm from the data, through minimization of the empirical risk. The set of parameter values constitutes the state of the model and suffices, together with its structure, to reproduce its predictions.
| Model class | Parameters |
|---|---|
| Linear or logistic regression | Variable coefficients, intercept |
| Decision tree | Split variables, thresholds, leaf values |
| Neural network | Connection weights, neuron biases |
In plain terms
The parameters are what the machine learned: the contents of the file saved at the end of training.
Point of caution
Parameters have no meaning independent of the feature encoding that produced them. A coefficient of 3,500 means one thing when the input is in square meters and another when it is in square feet. Serializing a model without serializing its preprocessing produces a file that predicts confidently and wrongly.
Rigorous definition
A loss function L : Y × Y → R+ maps a pair (observed value, predicted value) to
a positive scalar measure of their discrepancy, equal to zero if and only if the
prediction coincides with the observation, and increasing with the severity of
the gap. It must be computable and, for gradient-based algorithms, differentiable
with respect to the parameters.
Status
The loss is the operational definition of "predicting well". It is an engineering choice, not a given of the problem: two different loss functions applied to the same sample produce two different models.
In plain terms
The grading rule: what counts as a mistake, and how much that mistake costs.
Point of caution
The loss used for optimization and the metric used for business evaluation are two distinct objects, and they rarely coincide (chapters 029 and 052). Reporting the training loss to a business audience communicates nothing they can act on.
Rigorous definition
The arithmetic mean of the individual losses over the sample:
R̂(h) = (1/n) · Σ L(yi, h(xi)). This is the quantity the learning algorithm
actually minimizes.
Distinction from expected risk
The empirical risk is a mean over the n available observations; the expected
risk is an expectation over the generating distribution. The first is computable
and serves as the optimization objective; the second is the real objective and is
never directly accessible.
Point of caution
A zero empirical risk carries no information. It is always reachable by a
sufficiently expressive model, through simple memorization of the sample. A
report stating that training error reached zero states that the model is
expressive enough to memorize n points, and nothing else.
| Loss function | Problem type | Expression | Property |
|---|---|---|---|
| Squared error | Regression | (y − ŷ)² | Penalizes large gaps heavily; sensitive to outliers |
| Absolute error | Regression | absolute value of y − ŷ | Proportional penalty; robust to outliers |
| Binary cross-entropy | Binary classification | −[y·log(p) + (1−y)·log(1−p)] | Operates on the predicted probability p, not on the class |
| Hinge loss | Margin classification | max(0, 1 − y·score) | Also penalizes correct points that sit too close to the boundary |
Squared error fits the conditional mean; absolute error fits the conditional median. The choice is a business question, not a mathematical one: is the cost of a large error proportional to its size, or more than proportional? A logistics operation where a two-hour delay costs twice a one-hour delay calls for absolute error. A structural tolerance where a large deviation is catastrophic and a small one is harmless calls for squared error.
| Criterion | Parameter | Hyperparameter |
|---|---|---|
| Who sets the value | The learning algorithm | The engineer, or a search procedure |
| When | During training | Before training |
| Source of the value | The training data | A validated methodological choice |
| Examples | Coefficients, tree thresholds, network weights | Maximum depth, learning rate, regularization coefficient, number of trees |
| Effect of a change | Changes the predictions of the fitted model | Changes which model will be fitted |
Hyperparameters define H and the optimization procedure; parameters designate
the point selected within H. Their tuning is covered in chapter 009.
The cycle contains no step of understanding, interpretation, or causal reasoning. It contains a functional form chosen by the engineer, a discrepancy measure chosen by the engineer, and a numerical procedure that reduces that measure.
There is no understanding. There is iterative minimization of a scalar quantity.
This is not a rhetorical precaution. It has a direct operational consequence: a model will exploit any statistical regularity present in the data, including regularities with no business meaning, regularities that are artifacts of collection, and regularities that will not survive into production. The mechanism has no way to distinguish a causal signal from a coincidence that happens to reduce the loss.
An operator has an instrument with two dials and a graduated gauge that displays a deviation. The operator cannot hear the sound. He turns the first dial, reads the gauge, sees the deviation drop, and keeps turning the same way. When the deviation stops dropping, he moves to the second dial. After a few dozen adjustments the gauge shows a minimal deviation.
The operator heard nothing, knows no music theory, and could not explain what a perfect fifth is. He minimized the reading on a gauge.
Two consequences carry over exactly. If the gauge is miscalibrated, the instrument will be out of tune while the display reads zero — this is the critical role of the loss function and of the labels. If the gauge measures only one string out of six, the other five stay out of tune — this is the critical role of metric selection.
Task : estimate the sale price of a home from its floor area.
X = area in square meters, Y = price in dollars, n = 4.
| House | area_sqm | observed_price |
|---|---|---|
| A | 75 | $272,500 |
| B | 100 | $370,000 |
| C | 125 | $467,500 |
| D | 150 | $560,000 |
The engineer postulates an affine relationship price = a × area + b. That
choice defines H: the set of straight lines in the plane. The parameters to be
determined are a (price per square meter) and b (constant term). The loss
chosen is absolute error, for readability of the arithmetic, so the empirical
risk is the mean absolute error over the four houses.
Throughout, signed error is defined as observed − predicted: a negative value
means the model predicted too high.
| House | area_sqm | observed_price | predicted_price | absolute_error |
|---|---|---|---|---|
| A | 75 | $272,500 | $75,000 | $197,500 |
| B | 100 | $370,000 | $100,000 | $270,000 |
| C | 125 | $467,500 | $125,000 | $342,500 |
| D | 150 | $560,000 | $150,000 | $410,000 |
| Mean error | $305,000 |
Every prediction falls below the observed price, and the gap grows with area. The
coefficient a is too small. Direction of correction: increase a.
| House | area_sqm | observed_price | predicted_price | absolute_error |
|---|---|---|---|---|
| A | 75 | $272,500 | $150,000 | $122,500 |
| B | 100 | $370,000 | $200,000 | $170,000 |
| C | 125 | $467,500 | $250,000 | $217,500 |
| D | 150 | $560,000 | $300,000 | $260,000 |
| Mean error | $192,500 |
The mean error drops from $305,000 to $192,500: the correction went the right way. The diagnosis is unchanged, so the same move is repeated with a larger step.
| House | area_sqm | observed_price | predicted_price | absolute_error |
|---|---|---|---|---|
| A | 75 | $272,500 | $262,500 | $10,000 |
| B | 100 | $370,000 | $350,000 | $20,000 |
| C | 125 | $467,500 | $437,500 | $30,000 |
| D | 150 | $560,000 | $525,000 | $35,000 |
| Mean error | $23,750 |
The diagnosis now changes in nature. The errors are of comparable magnitude and
all of the same sign: predictions still sit below observed prices, by roughly
$25,000. A constant offset like that cannot be fixed by a — increasing it would
degrade the large areas. It belongs to the constant term b.
| House | area_sqm | observed_price | predicted_price | signed_error | absolute_error |
|---|---|---|---|---|---|
| A | 75 | $272,500 | $287,500 | −$15,000 | $15,000 |
| B | 100 | $370,000 | $375,000 | −$5,000 | $5,000 |
| C | 125 | $467,500 | $462,500 | +$5,000 | $5,000 |
| D | 150 | $560,000 | $550,000 | +$10,000 | $10,000 |
| Mean error | $8,750 |
The errors now carry opposite signs: two predictions too high, two too low.
No global change to a or to b can reduce all four simultaneously — improving
one house degrades another. The process has reached a regime of compromise, and
that is the stopping criterion.
estimated_price = 3,500 × area_sqm + 25,000Those two numbers are the entirety of what was learned. Applied to a new case, a 110 square meter home absent from the sample:
estimated_price = 3,500 × 110 + 25,000 = $410,000The model produces a value for an observation it never encountered. That step from the known to the unknown is generalization, treated in section 7.
First lesson — No domain understanding is involved
The procedure drew on no knowledge of the housing market. It consulted no price
schedule; it has no notion of neighborhood, of the condition of the property, or
of interest rates. It produced two numbers minimizing a mean of gaps. That
a = 3,500 reads as a price per square meter is an interpretation performed by
the analyst after the fact, not knowledge held by the model.
Second lesson — The mechanism is try, measure, adjust
Four iterations were shown; a real algorithm performs thousands, with small-amplitude adjustments guided by the gradient rather than by inspecting the signs of the errors. The logical structure is exactly the one in section 3, with nothing added: propose, measure the gap, correct in the direction that reduces the gap.
Third lesson — The form of the rule is chosen by the engineer
The machine did not decide that price would be an affine function of area. That assumption was made before any computation and constrains the outcome permanently. If the true relationship is a step function, or plateaus beyond a certain area, or depends on an interaction with neighborhood, this model can never represent it, however much data is supplied. Choosing the model class is an engineering act prior to and above the fitting: it is exactly the approximation error defined in section 1.
Task : predict whether a prospect will make a purchase, from age and annual
salary. X = (age, salary), Y = {Purchase, No purchase}, n = 6.
| Prospect | age | annual_salary | purchased |
|---|---|---|---|
| P1 | 22 | $28,000 | No |
| P2 | 25 | $32,000 | No |
| P3 | 31 | $41,000 | No |
| P4 | 38 | $58,000 | Yes |
| P5 | 45 | $72,000 | Yes |
| P6 | 52 | $85,000 | Yes |
The formal difference from section 4 lies entirely in the nature of Y: a
finite unordered set replaces an interval of real numbers. That single difference
changes the loss function, the evaluation metrics, and the geometric reading of
the model.
Rigorous definition
For a classifier f̂ : X → Y, the decision boundary is the subset of X at which
the decision rule changes the class it assigns. In the binary case built on a
continuous score s(x) and a threshold t, it is the level set
{ x ∈ X | s(x) = t }. It partitions X into decision regions, each associated
with one class.
Geometric property
In a space of d variables the boundary is a hypersurface of dimension d − 1:
a point on a line, a curve in the plane, a surface in three dimensions.
| Model class | Shape of the boundary |
|---|---|
| Logistic regression | Hyperplane: a straight line in the plane |
| Decision tree | Axis-parallel segments, a staircase |
| Kernel support vector machine | A smooth curve whose form depends on the kernel |
| Neural network | An arbitrarily complex hypersurface |
Point of caution
The boundary is movable without retraining: changing the threshold t slides
it along the level sets of the score. This property is exploited in chapter 062.
On these six observations an affine rule separates the two groups perfectly:
raw_score = salary + 1,000 × age, with the boundary at 90,000.
| Prospect | raw_score | Position | predicted_class | observed_class |
|---|---|---|---|---|
| P1 | 50,000 | Below the boundary | No | No |
| P2 | 57,000 | Below the boundary | No | No |
| P3 | 72,000 | Below the boundary | No | No |
| P4 | 96,000 | Above | Yes | Yes |
| P5 | 117,000 | Above | Yes | Yes |
| P6 | 137,000 | Above | Yes | Yes |
Point of caution : perfect separation on six observations is not a performance. When the number of observations is small relative to the number of variables, perfect separability is common and is itself a warning sign for overfitting (chapter 031). Six points in two dimensions can be separated by a line far more often than the underlying phenomenon warrants.
A classifier does not produce a class directly. It first produces a continuous score, usually transformed into an estimate of a conditional probability, and then applies a threshold to decide.
Rigorous definition
A probabilistic classifier estimates the conditional law P(Y | X = x). For a
binary problem with Y = {0, 1} it produces p̂(x) ∈ [0, 1] estimating
P(Y = 1 | X = x). The decision rule associated with a threshold t is: ŷ = 1
if p̂(x) ≥ t, and ŷ = 0 otherwise.
| Act | Nature | Who determines it | Changeable after training |
|---|---|---|---|
Estimating p̂(x) | Statistical | The model, by fitting on D | No, not without retraining |
Choosing the threshold t | Decisional | The organization, according to error costs | Yes, immediately |
Point of caution on calibration
A high score is not a reliable probability. A model is calibrated when, among the observations to which it assigns a score of 0.80, approximately 80% do belong to the positive class. The raw scores of many algorithms lack this property and require recalibration (chapter 029).
In plain terms
The model produces a degree of confidence. Turning that degree into a decision is a management choice, not a technical one. The value 0.50 is an implementation default, never a justification.
For the new case "age 35, salary $60,000" the raw score is 95,000, which is 5,000
above the boundary. Transformed into a probability, it gives
P(y = Purchase | x) = 0.81, and the default threshold of 0.50 leads to
predicting "Purchase". Raise the threshold to 0.85 and the same observation, the
same model, and the same score lead to predicting "No purchase". The model did
not change; the decision changed. Where to set the threshold depends on the
relative cost of the two error types, treated in chapters 052 and 062.
| Criterion | Regression (section 4) | Classification (section 5) |
|---|---|---|
Nature of Y | An interval of real numbers | A finite unordered set |
| Raw output of the model | A numeric value | A score or a probability per class |
| Extra decision step | None | Applying a threshold |
| Usual loss function | Squared error, absolute error | Cross-entropy |
| Geometric reading | A fitted curve | A decision boundary |
| Evaluation metrics | Mean error, coefficient of determination | Accuracy, precision, recall, area under the curve |
| Training | Evaluation | Production | |
|---|---|---|---|
| Nature of the data | Training set, historical | Test set, historical and held out | Live flow, new observations |
| Label available | Yes, used for fitting | Yes, for comparison only | No, not at prediction time |
| What the system does | Adjust the parameters | Predict and measure the gap | Predict |
| Code call | model.fit(X_train, y_train) | model.predict(X_test) then compare to y_test | model.predict(x_new) |
| Is the model modified | Yes | No | No |
| Analogy | Studying with the answer keys | The mock exam on unseen problems | Real professional practice |
| What you obtain | A fitted model | An estimate of the expected risk | An operational decision |
# Regime 1 - training: the model sees both x and y
model.fit(X_train, y_train)
# Regime 2 - evaluation: the model sees x; y is used only for comparison
y_pred = model.predict(X_test)
score = metric(y_test, y_pred)
# Regime 3 - production: y does not exist yet
y_estimated = model.predict(x_new_observation)The professional difficulty is not in writing those three lines. It is in
building X_train, X_test, and their labels correctly.
Major point of caution
Evaluating a model on the data used to train it produces an optimistic estimate with no informational value.
The formal justification was given in section 1: f̂ was selected precisely to
minimize R̂ on those observations, so R̂(f̂) is a biased estimator of R(f̂).
Splitting strategies and cross-validation are covered in chapters 026 and 034.
Rigorous definition
D used to fit the parameters.Why validation and test are separated
Any choice made in view of a score on a dataset implicitly optimizes on that dataset. A validation set consulted several dozen times stops being neutral: the selection process has introduced an optimistic bias into it, of the same nature as the bias on the training set, though smaller in magnitude.
Operational rule
The test set is opened once, and the number obtained is the number reported. If it is consulted and then the model is modified, it has changed nature and become a validation set. Nothing in the code enforces this; it is a discipline.
A school wants to estimate how its students will do on the final exam.
Faulty protocol : the mock exam reuses the problems from the practice workbook. The results are excellent and predict nothing; they measure how well the workbook was memorized.
Valid protocol : the mock exam uses unseen problems of the same nature and difficulty as the final. The results are lower, and they are usable as an estimate.
The difference between the two protocols is not the difficulty of the problems. It is whether or not those problems were used during preparation. That is the only question that governs the validity of an evaluation.
Rigorous definition
Generalization is the ability of a fitted model to maintain a given level of
performance on observations drawn from the same distribution P(X, Y) as the
training sample but not observed during fitting. It is quantified by the
generalization gap R(f̂) − R̂(f̂): a small gap indicates that the performance
measured on the sample is representative of the performance to expect.
Validity condition
The definition carries an explicit condition: "drawn from the same distribution". When the distribution of production data differs from that of the training data, the guarantee no longer applies. That phenomenon is drift, treated in chapter 082; generalization offers no protection against it.
In plain terms
To generalize is to work on cases never encountered, not only on the cases used for learning.
| Training error | Generalization error | |
|---|---|---|
| Definition | Empirical risk on the training set | Expected risk over the generating distribution |
| Notation | R̂(f̂) on D_train | R(f̂) |
| Computable | Yes, directly | No, only estimable |
| Practical estimator | — | The error measured on the test set |
| What it measures | Quality of the fit to the data seen | Expected quality in production |
| Can it be zero | Yes, with a sufficiently expressive model | No, irreducible error bounds it |
| Decision value | None, taken alone | The only decision value in the project |
The gap between the two is the practitioner's primary diagnostic instrument.
| Training error | Test error | Diagnosis | Treatment |
|---|---|---|---|
| High | High | Underfitting: the model is too constrained | Enrich the variables, increase expressiveness |
| Low | High | Overfitting: the model memorized | Regularize, simplify, increase the volume |
| Low | Low | Satisfactory regime | Verify there is no leakage (chapter 028) |
| High | Low | Abnormal situation | Check the splitting protocol |
The fourth row deserves a comment. A test error below the training error is almost never good news. It usually means the split was not what it appears to be — an easier test subset, a duplicated preprocessing step, or a stratification that concentrated the hard cases on one side.
A demand forecasting model is trained on data from January through June. It reaches a mean error of 3.1% over that period. That figure carries no decision value: it describes the model's ability to re-fit a period whose values it already knows.
The only question that matters is: what is the error on July?
July was not observed during fitting. Performance on July is the only information that says anything about what the model will produce in August and in every period to come. What holds for a temporal split holds for a random split, a split by site, by customer, or by region. The question is invariant: what is the performance on what was not used for fitting?
None of these techniques has a purpose of its own. Data splitting does not exist to satisfy a convention; regularization does not exist to produce more elegant models. Each exists to make the generalization error estimable, or to reduce it.
A practitioner who cannot trace every methodological choice back to the generalization gap is applying recipes.
An actor knows the full text of the play. He delivers it flawlessly, with the intonations set in rehearsal. On that repertoire his performance is perfect.
On opening night a fellow actor skips a line. The actor who memorized stops: the learned sequence no longer matches the situation. The actor who understood the dramatic situation improvises a coherent line and carries on. The two were indistinguishable in rehearsal — they scored identically on the training set.
| Actor | On the known text | On the unforeseen situation | Corresponding model |
|---|---|---|---|
| The one who memorized | Perfect | Fails | Overfitting |
| The one who understood | Good | Good | Generalization |
Transposition : the only way to tell the two actors apart is to place them in front of an unrehearsed situation. That is exactly the function of the test set. Without that trial the two profiles are indistinguishable — and the failing profile posts the better apparent scores.
Context. An industrial operator runs 300 instrumented machines across four sites, streaming continuous readings of vibration, temperature, power draw, and usage counters. The maintenance department holds an intervention log covering thirty months. An unplanned outage costs roughly $18,000 in lost production; a planned preventive intervention costs roughly $1,200.
| Condition | Verification performed | Status |
|---|---|---|
| An observable target exists in the history | The log allows every failure to be dated | Met |
| The history is sufficient in volume and in events | 30 months, 300 machines, 412 unplanned failures | Met |
| The variables are available at prediction time | Time-stamped readings, accessible in real time | Met |
| The prediction leads to an actionable decision | A preventive intervention can be scheduled | Met |
| The cost of error is acceptable | False positive $1,200, false negative $18,000 | Met, with strong asymmetry |
Every one of these five conditions is checked before any modeling work. A failure on any single one invalidates the project regardless of how the other four look.
The initial business phrasing — "anticipate failures" — is not usable as it stands. Four candidate formulations were examined.
| Candidate formulation | Problem type | Disqualifying defect |
|---|---|---|
| "Is the machine currently failed?" | Classification on instantaneous state | The variable is directly observable and the failure has already happened. No anticipation, therefore no possible decision |
| "What is the remaining useful life?" | Regression | Requires the exact future failure date for every observation. Machines still in service have no end date: the sample is censored, which invalidates a naive regression |
| "Will the machine fail one day?" | Classification with no horizon | Degenerate target: the answer is positive for every machine. Prevalence of 100%, nothing to learn |
| "Will the machine fail within the next 7 days?" | Binary classification with a fixed horizon | Retained. Residual defects known and manageable: strong class imbalance, sensitivity to the choice of horizon |
Target retained : failure_7d = 1 if an unplanned failure occurs on the
machine within 7 days following the date of the reading, 0 otherwise.
| Horizon considered | Consequence |
|---|---|
| 2 days | More accurate predictions but unusable: the part does not arrive in time. No business value |
| 7 days | Compatible with the intervention supply chain. Horizon retained |
| 30 days | The precursor signal is too weak at that range: prevalence rises but separability collapses. Too many alerts, too early |
Principle : the horizon of a predictive target follows from the lead time required to act on the decision, never from statistical convenience.
| Variable | Description | Business justification |
|---|---|---|
vib_rms_24h | Mean RMS vibration over 24 hours | Direct indicator of bearing wear and imbalance |
vib_delta_7d | Deviation from the mean of the preceding 7 days | Degradation shows up as a change, not as an absolute level specific to each machine |
temp_max_24h | Maximum housing temperature over 24 hours | Abnormal heating signals friction or a lubrication fault |
temp_dev_setpoint | Deviation from the nominal temperature for the model | Normalizes temperature across machine generations |
hours_since_maintenance | Operating hours since the last intervention | Wear is a function of service time, not calendar time |
mean_daily_cycles_7d | Mean number of daily cycles over 7 days | Measures the intensity of use |
power_draw_deviation | Deviation of power consumption from the machine's baseline | Over-consumption at constant load indicates increased mechanical resistance |
short_stops_30d | Number of micro-stoppages over 30 days | Micro-stoppages frequently precede major failures |
machine_age_months | Age of the machine | Failure rate depends on position in the life cycle |
site | Operating site | Captures environmental differences: ambient temperature, air quality, local maintenance practices |
Point of caution : vib_delta_7d, temp_dev_setpoint, and
power_draw_deviation are constructed variables, not raw readings. Their
construction must be reproducible identically in production (chapter 024). A
rolling mean computed over the full history in the notebook and over the
trailing window in production is not the same variable.
One observation is a (machine, date) pair, giving roughly 270,000 rows. The rate of positive events is about 1.1%.
| machine_id | date | vib_rms_24h | vib_delta_7d | temp_max_24h | hours_since_maintenance | short_stops_30d | site | failure_7d |
|---|---|---|---|---|---|---|---|---|
| M-014 | 2024-03-11 | 2.1 | +0.1 | 64 | 820 | 0 | North | 0 |
| M-014 | 2024-03-12 | 2.2 | +0.2 | 65 | 828 | 0 | North | 0 |
| M-087 | 2024-05-02 | 5.8 | +2.4 | 81 | 2,340 | 4 | South | 1 |
| M-087 | 2024-05-03 | 6.1 | +2.7 | 84 | 2,348 | 5 | South | 1 |
| M-201 | 2024-07-19 | 3.4 | +0.3 | 71 | 1,510 | 1 | East | 0 |
| M-233 | 2024-09-08 | 6.2 | +3.1 | 88 | 2,780 | 7 | North | 1 |
A shallow decision tree fitted on this data produces the structure below. The probabilities are those of the positive class within each leaf.
How to read it : the most discriminating variable is not the vibration level but its recent change — a result consistent with domain expertise, since every machine has its own vibration signature and it is the deviation from that signature that carries the information. This consistency is not a validation, but its absence would have been a warning about how the data was built.
| Pitfall | How it appears in this case | Consequence if untreated | Chapter |
|---|---|---|---|
| Class imbalance | 1.1% positive observations | A model that always predicts "no failure" reaches 98.9% accuracy and detects nothing. Accuracy is unusable as a metric here | 050 |
| Asymmetric error cost | False negative $18,000, false positive $1,200, a ratio of 15 to 1 | A threshold of 0.50 minimizes the number of errors, not their cost. The threshold must be lowered to the economic break-even point | 052 and 075 |
| Data leakage | The columns intervention_date, failure_code, and downtime_hours are populated only after the failure | Near-perfect validation performance, collapse in production. The most frequent failure mode on this type of project | 028 |
| Overfitting | 270,000 rows but only 412 real events, and strong correlation between successive readings from one machine | The model memorizes the signatures of the machines that failed instead of learning the degradation mechanism | 031 |
| Drift | Fleet renewal, a change of parts supplier, modified production rates | Performance degrades gradually with no alert, since the sensors keep emitting plausible values | 082 |
What happens
Drawing 20% of the 270,000 rows at random for the test set places machine M-087's reading from May 2 in training and its reading from May 3 in test, when the two are nearly identical. The model is then evaluated on observations whose near-duplicates it saw while fitting: the test error becomes an optimistic estimator of the generalization error. The condition "not observed" is satisfied formally at the row level and violated in substance.
Correct protocol
The split must reproduce the production situation: the model will be applied to dates later than those it was trained on, and sometimes to machines installed after training.
| Split | Question it answers |
|---|---|
| Temporal: train on 24 months, test on the last 6 | Does the model work over the period to come? |
| By machine: train on 240 machines, test on 60 held out | Does the model work on a machine never observed? |
Both estimates are useful and they answer different questions (chapter 026).
The example in section 4 produced a = 3,500. It is tempting to conclude that
the model "knows" a square meter is worth $3,500. The model holds no notion of a
square meter, of price, or of housing: it holds two numbers that minimize a mean
of gaps over four observations.
Operational consequence : an area of 900 square meters will produce an estimate of $3,175,000, with no mechanism raising any alarm about the absurdity of the extrapolation.
Correct formulation : "The model fitted two parameters that minimize the mean gap to the observed prices on the sample provided."
Supervision refers to the availability of reference labels during training. It says nothing about the operating regime of the deployed system, which may be fully automatic. The confusion leads teams to assume a human control that does not exist, and therefore to omit the safeguards that would be needed.
Correct formulation : "The learning is called supervised because every training observation came with the target value to reproduce. Whether an operator is present in production is a separate decision."
The label is presented as ground truth. It is in fact the product of a process — recording, self-reporting, judgment, rule — that has its own biases, its own blind spots, and its own error rate.
Characteristic case : training a model on detected fraud teaches it to detect what the current system already detects. The frauds that slip past that system are labeled "not fraudulent" in the data, and the model learns them as legitimate.
Correct formulation : "The model's performance is bounded by the quality of the process that produced the labels. The first step of the project is to investigate that process."
"Predict failures", "predict churn", "predict fraud" are not targets but intentions. A usable target has three components: an unambiguously defined event, a time horizon, and an observation date from which that horizon runs. Without them the target column cannot be built, and two people working on the same project will build two different versions of it.
Correct formulation : "failure_7d equals 1 when an unplanned failure occurs
within 7 days following the date of the reading, and 0 otherwise."
The score obtained on the data used for fitting is an optimistically biased estimator of the expected risk: the model was selected to minimize precisely that quantity. An unconstrained decision tree reaches 100% accuracy on its training data whatever the data happens to be; that figure measures its memorization capacity and nothing else.
Correct formulation : "The reported error is the one measured on a test set made of observations that took part neither in fitting the parameters nor in selecting the hyperparameters."
Parameters are determined by the algorithm from the data; hyperparameters are set before training and define both the hypothesis space and the optimization procedure.
Operational consequence : hyperparameters cannot be tuned on the training set, because they govern the model's ability to fit that set. Tuning them there leads systematically to the most expressive model available, and therefore to overfitting (chapter 009).
Correct formulation : "Hyperparameters are selected by validation on data distinct from the data used to fit the parameters."
A variable present in the history is not necessarily available at the moment the prediction has to be produced: the maintenance log contains downtime hours, which are known only after the failure.
Test to apply systematically : for every candidate variable, ask whether its value will be known at the exact instant the model must predict in production. Any negative answer disqualifies the variable. The symptom is abnormally high validation performance followed by collapse in production. That is data leakage, treated in chapter 028.
Correct formulation : "Every retained variable is available at prediction time, verified variable by variable with the operators of the source system."
The definition of generalization carries the condition "drawn from the same distribution". That condition is verifiable at evaluation time; it is not guaranteed afterwards. A model validated at 92% accuracy can be running at 71% eighteen months later with no technical error having occurred and no alarm having fired.
Correct formulation : "The measured performance holds for the distribution observed over the evaluation period. Drift monitoring and a retraining procedure are planned (chapter 082)."
FORMALIZATION
Data : D = {(xi, yi)}, i = 1..n, drawn from P(X, Y), unknown and fixed
Assumption : there exists f : X -> Y, possibly noisy
Objective : minimize the expected risk R(h) = E[L(y, h(x))]
Reality : minimize the empirical risk R^(h) = (1/n) SUM L(yi, h(xi))
Principle : empirical risk minimization (ERM)
THE WORD "SUPERVISED"
Denotes the availability of a reference label during training.
Denotes no human oversight at run time.
LABELS
Six sources: natural history, user action, human annotation,
domain expert, automatic business rule, instrument measurement.
Mandatory question: how was the label produced?
Labels from a rules engine: performance is capped there.
THE MECHANISM
Try -> Measure the error -> Correct -> Try ...
Parameter : fitted by the algorithm, from the data
Hyperparameter : set by the engineer, before training
There is no understanding. There is iterative minimization.
THE TWO PROBLEM TYPES
Regression : Y continuous -> a value
Classification : Y finite -> a score, then a threshold, then a class
The decision boundary moves when the threshold changes,
with no retraining.
THE THREE REGIMES
Training : y known and used model.fit(X_train, y_train)
Evaluation : y known, hidden model.predict(X_test) then compare
Production : y unknown model.predict(x_new)
THE SOLE OBJECTIVE
Generalization gap = expected risk - empirical risk
Training error : no decision value
Test error : the only usable estimate
Data splitting, cross-validation, regularization, overfitting
diagnosis: none has a purpose of its own.
All of them serve generalization.Summary statement
Supervised learning fits a parameterized function on a sample of labeled observations by minimizing an empirical risk that is only a computable stand-in for the risk actually targeted; the value of the model is never measured on the observations used to fit it, but on the ones it has not seen.
Associated quizzes :
004.1-quiz-formalization.md,
004.2-quiz-labeled-data.md,
004.3-quiz-fitting-mechanism.md,
004.4-quiz-worked-regression.md,
004.5-quiz-classification-boundary.md,
004.6-quiz-three-regimes.md,
004.7-quiz-generalization.md
Next chapter : 005.0-dataset-structure.md