Parameters and Hyperparameters

Block 1 — Core vocabulary

Objective

separate without hesitation what a model learns from what an engineer imposes on it, name the usual hyperparameters of each algorithm family and state the effect of each one, verify that separation directly in code, and acquire the evaluation vocabulary developed in the later blocks — loss, metric, score, decision threshold, naive baseline.

Estimated duration

45 minutes

Prerequisites

chapters 001 to 008

Associated quizzes

009.1-quiz-learned-parameter.md to 009.7-quiz-loss-metric-score-threshold-baseline.md

1. Two categories of numbers inside a model

1.1 The distinction

A trained model is a numeric object: it holds numbers. Those numbers are not all of the same nature, and confusing them is one of the most frequent errors early in practice.

Chapter 008 separated the algorithm, which is a procedure, from the model, which is the result of applying that procedure to a dataset. The relationship between the two categories of numbers follows from that separation directly: hyperparameters configure the procedure, parameters constitute its result.

How to read the diagram. Hyperparameters appear in the final model on the same footing as the parameters, since they describe its configuration, but they arrive there by a different route: they never passed through the optimization procedure. Two arrows reach the model, and only one of them crosses the data.

1.2 Why the confusion is expensive

The distinction is not a matter of vocabulary hygiene. Four concrete failures follow from collapsing the two categories, and each one is common enough to be worth naming.

A reproducibility failure. A model is reproducible when the algorithm, the hyperparameters, the data and the random seed are all recorded. A team that records "we used a gradient boosting model" has recorded the algorithm and nothing else. The learning rate and the number of iterations are not recoverable from the serialized artifact by inspection alone, and a retraining run six months later produces a different model with no way to tell why.

A tuning failure. Parameters cannot be tuned, and hyperparameters cannot be learned. An analyst who tries to "adjust the coefficients until the recall improves" is hand-fitting the training data through a very narrow interface; an analyst who expects fit() to discover the right tree depth will never get it, because depth is a bound on the search, not an outcome of it.

A leakage failure. Preprocessing statistics — imputation medians, encoder categories, scaling means — are learned quantities (section 4.3). Treating them as configuration invites computing them on the full dataset before the split, which transfers information from the test set into the training set (chapter 028).

A communication failure. "The model found that a bill above $65 predicts churn" and "we set the threshold at $65" describe opposite situations. The first is a finding to be validated; the second is an assumption to be defended. A stakeholder who cannot tell which one they were given cannot challenge either.

2. The parameter: what the optimization procedure adjusts

2.1 Linear models: coefficients and intercept

Multiple linear regression models the target as an affine combination of the explanatory variables:

ŷ = b₀ + b₁·x₁ + b₂·x₂ + ... + b_p·x_p

The learned parameters are the p coefficients b₁ to $b_p$ and the intercept b₀, that is p + 1 values. Nothing else is stored.

Worked figures — estimating the price of a home. The regression below is fitted on 1,200 simulated transactions described by four explanatory variables, with price in dollars.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(11)
n = 1_200

area_sqm    = np.round(np.clip(rng.normal(88, 26, n), 25, 220), 1)
room_count  = np.clip(rng.poisson(1.4, n) + 1 + (area_sqm > 100).astype(int), 1, 7)
age_years   = rng.integers(0, 61, n)
distance_km = np.round(np.clip(rng.gamma(2.2, 2.4, n), 0.2, 30), 2)

price = (42_000 + 2_180 * area_sqm + 6_400 * room_count
         - 870 * age_years - 3_150 * distance_km + rng.normal(0, 14_000, n))

H = pd.DataFrame({"area_sqm": area_sqm, "room_count": room_count,
                  "age_years": age_years, "distance_km": distance_km})

reg = LinearRegression().fit(H, price)
print("intercept_:", round(float(reg.intercept_), 1))
print(pd.Series(np.round(reg.coef_, 1), index=H.columns).to_string())

listing = pd.DataFrame([{"area_sqm": 85.0, "room_count": 3,
                         "age_years": 12, "distance_km": 4.0}])
print("prediction:", round(float(reg.predict(listing)[0]), 1))

intercept_: 37756.4
area_sqm       2197.4
room_count     6914.4
age_years      -834.4
distance_km   -3146.9
prediction: 222676.8

The whole model fits in those five numbers. For a home of 85 m², three rooms, twelve years old, four kilometers from the center, evaluating the equation by hand on the rounded coefficients gives 37,756.4 + 2,197.4×85 + 6,914.4×3 − 834.4×12 − 3,146.9×4 = $222,678, against $222,677 returned by predict. The gap of one dollar is the rounding applied for display, and it is worth noticing: the coefficients printed in a report are never quite the coefficients the model computes with.

Point of caution. The "all else being equal" clause assumes the variables can move independently of one another, which they rarely can. Here room_count was constructed to depend on area_sqm, and the fitted coefficients split the joint effect between them in a way that has no stable interpretation. Multicollinearity is covered in chapter 017.

In logistic regression the structure is identical, the linear combination being passed through a sigmoid to produce a probability. The parameters remain the coefficients and the intercept (chapter 036).

2.2 Neural networks: weights and biases

Worked figures — a multilayer perceptron for a churn score. Architecture: 20 input variables, two hidden layers of 64 then 32 neurons, one output.

The count is verifiable against the fitted object rather than taken on trust:

from sklearn.neural_network import MLPClassifier

mlp = MLPClassifier(hidden_layer_sizes=(64, 32), max_iter=8, random_state=0)
mlp.fit(X20, y)

weights = sum(a.size for a in mlp.coefs_)
biases = sum(a.size for a in mlp.intercepts_)
print("weights:", weights, "biases:", biases, "total:", weights + biases)
print("shapes:", [a.shape for a in mlp.coefs_], [a.shape for a in mlp.intercepts_])

weights: 3360 biases: 97 total: 3457
shapes: [(20, 64), (64, 32), (32, 1)] [(64,), (32,), (1,)]

This network learns 3,457 numbers. The number of layers and their widths — that is, hidden_layer_sizes=(64, 32) — are not learned: they are hyperparameters, and they are what determines how many parameters there are to learn. A hyperparameter therefore governs the quantity of parameters, which is the direct link to the notion of capacity developed in section 7.

2.3 Decision trees: structure, split variables and thresholds

A decision tree has neither coefficients nor weights. What it learns is combinatorial in nature:

• for each internal node, the variable to split on;
• for each internal node, the threshold on that variable;
• the topology of the tree, that is, which nodes get split and which become leaves;
• for each leaf, the predicted value or the class distribution.

Partial view: the tree actually obtained in section 5.3, with the right subtree abbreviated.

The value 57.50 was not chosen by an analyst. The algorithm evaluated the candidate splits on every variable and kept the one that reduced the node's impurity most. It is learned, exactly as a regression coefficient is (mechanism detailed in chapter 037). What the engineer fixed is the frame of that search: maximum depth, minimum leaf size, impurity criterion.

2.4 Support vector machines

An RBF-kernel SVM fitted on the 4,000 standardized observations of section 5 retains 2,307 support vectors, split 1,167 for the negative class and 1,140 for the positive one — that is 58% of the training set. Nobody asked for that number; it fell out of the dual optimization.

2.5 The case of memorization methods

2.6 Orders of magnitude

Every tree, forest and SVM figure in this table was measured on the dataset of section 5, not estimated. Taken together the rows make one central point: the number of learned parameters is not a property of the algorithm alone. It is determined jointly by the hyperparameters and by the data.

3. The hyperparameter: what the engineer fixes before optimization

3.1 The usual hyperparameters, by algorithm family

Point of caution on C and alpha. These two hyperparameters govern the same thing — the strength of the regularization — but in opposite directions. alpha is proportional to the strength of the penalty; C is its inverse. Raising alpha regularizes more; raising C regularizes less. That inversion is a standing source of error in technical interviews, and section 5.5 shows it numerically.

3.2 The five functions of a hyperparameter

Hyperparameters do not all carry the same weight. Sorting them by function is what keeps tuning from being a random walk.

Capacity hyperparameters are decisive and are tuned first. Optimization hyperparameters matter mostly in boosting and in neural networks. Ensemble structure hyperparameters show sharply diminishing returns. Problem-handling hyperparameters become critical under class imbalance or asymmetric error costs. Execution hyperparameters have no effect on expected performance at all.

Point of caution on random_state. This argument is formally a hyperparameter: it is fixed before training and is not adjusted by the procedure. It must nonetheless never be included in a search. Selecting the seed that maximizes the validation score means exploiting sampling noise, not improving the model. The legitimate role of random_state is reproducibility.

4. The operational criterion of distinction, borderline cases and pitfalls

4.1 The single question

One question separates the two categories, whatever the library or the algorithm:

Is this quantity adjusted by the optimization procedure executed during fit(), or was it fixed before that procedure started?

The convention is verifiable in three lines, and the check is worth running once on any estimator you have not used before.

import pandas as pd
from sklearn.exceptions import NotFittedError
from sklearn.tree import DecisionTreeClassifier

df = pd.read_csv("telecom_churn_4k.csv")
y = df["churned"]
X = df.drop(columns="churned")

tree = DecisionTreeClassifier(max_depth=3, min_samples_leaf=50, random_state=0)
before = tree.get_params()

try:
    tree.feature_importances_
except NotFittedError as e:
    print("Before fit ->", type(e).__name__)

tree.fit(X, y)
after = tree.get_params()
print("Hyperparameters unchanged by fit:", before == after)
print("Learned attributes now available:",
      [a for a in ("feature_importances_", "tree_", "classes_", "n_features_in_")
       if hasattr(tree, a)])
print("Number of learned split nodes:", tree.tree_.node_count - tree.get_n_leaves())

Before fit -> NotFittedError
Hyperparameters unchanged by fit: True
Learned attributes now available: ['feature_importances_', 'tree_', 'classes_', 'n_features_in_']
Number of learned split nodes: 7

Interpretation. The two directions of the criterion are exhibited in one block. Before fit(), the hyperparameters are all readable and the learned attributes do not exist — the access raises rather than returning a default. After fit(), the learned attributes exist and the hyperparameter dictionary compares equal to the one captured beforehand. Training added information; it changed no setting.

4.2 Classifying concrete cases

4.3 The four borderline cases to master

Borderline case 1 — Preprocessor statistics. The mean and standard deviation of a StandardScaler are estimated from the data: they are parameters in the sense of the fit() procedure, even though they do not belong to the predictive model. The decisive consequence is procedural. They are estimated on the training set alone, then applied unchanged to the other sets. Estimating them on the full dataset is a leak (chapter 028).

Borderline case 2 — Early stopping. With early_stopping=True, the number of iterations retained is determined by the procedure itself, by watching the loss on an internal validation set. The boundary appears to blur; it does not. The decision to enable early stopping, the size of that internal set, the patience and the monitored metric all remain hyperparameters fixed in advance. The number of iterations retained is a result.

Borderline case 3 — Automatically optimized hyperparameters. A grid search determining max_depth does not turn it into a parameter. The search is an outer loop relaunching complete training runs, each with a value fixed in advance. A parameter is adjusted inside one training run; a hyperparameter is chosen between training runs (chapter 035).

Borderline case 4 — Derived parameters. feature_importances_ is not optimized directly: it is a statistic computed after the fact from the impurity reductions of the learned structure. It is nonetheless a learned quantity, since it does not exist before fit() and depends entirely on the data. The same holds for the coef_ of a linear SVM, recomposed from the dual coefficients.

4.4 Quantities that look like parameters and are not

Three families of numbers are read as learned quantities by mistake, in reports and in interviews alike.

Numbers that came out of exploratory analysis. A binning threshold chosen from a histogram, a winsorization cap set at the 99th percentile, a correlation cutoff used to drop variables — all of these are numbers derived from data, which makes them feel learned. They are hyperparameters of the pipeline, fixed by the analyst before the estimator runs. Their only correct status is as decisions to be documented and, where it matters, to be selected on validation like any other hyperparameter.

Numbers that came from the business. A minimum ticket size, a regulatory ceiling, an eligibility age. These are constants of the problem. They constrain the data or the decision, and no procedure will ever revise them.

Numbers a library printed. n_iter_, n_features_in_, n_support_, the elapsed fit time. These exist only after fit() and carry a trailing underscore in the scikit-learn case, but they play no part in computing a prediction. They diagnose the run. The decision tree at the top of section 4.1 separates them from true parameters with its second question.

5. Verification in code: get_params() against coef_ and feature_importances_

5.1 The demonstration dataset

Every figure in this section and the two that follow comes from the dataset built below. It describes 4,000 customers of a telecom operator through five explanatory variables, with a binary target churned. The generative model is deliberately simple — an affine logit passed through a Bernoulli draw — so that what the estimators recover can be compared against what was put in.

import numpy as np
import pandas as pd

rng = np.random.default_rng(96)
n = 4_000

tenure_months = rng.integers(1, 121, n)
monthly_bill  = np.round(np.clip(rng.normal(58, 14, n), 15, 95), 2)
support_calls = rng.poisson(1.6, n)
satisfaction  = np.clip(np.round(rng.normal(3.6, 1.1, n)), 1, 5)
avg_data_gb   = np.round(np.clip(rng.normal(28, 12, n), 0.5, None), 1)

logit = (0.58
         - 0.026 * tenure_months
         + 0.039 * monthly_bill
         + 0.460 * support_calls
         - 0.724 * satisfaction)
churned = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

pd.DataFrame({"tenure_months": tenure_months, "monthly_bill": monthly_bill,
              "support_calls": support_calls, "satisfaction": satisfaction,
              "avg_data_gb": avg_data_gb, "churned": churned
              }).to_csv("telecom_churn_4k.csv", index=False)

Two properties of this construction matter for what follows. avg_data_gb was generated independently of the target: it carries no signal whatsoever, and its fitted coefficient is the control case in every experiment below. And the target is a genuine Bernoulli draw, not a deterministic function of the features, so no honest model can classify the training set perfectly. Section 5.5 shows one doing exactly that anyway.

df = pd.read_csv("telecom_churn_4k.csv")
y = df["churned"]
X = df.drop(columns="churned")
print("shape:", X.shape, "| positive rate:", round(float(y.mean()), 4))

shape: (4000, 5) | positive rate: 0.3835

5.2 The hyperparameters: get_params()

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

Xs = pd.DataFrame(StandardScaler().fit_transform(X), columns=X.columns)

lr = LogisticRegression(C=1.0, max_iter=1000, random_state=0)
lr.fit(Xs, y)

for k, v in sorted(lr.get_params().items()):
    print(f"{k}: {v!r}")

C: 1.0
class_weight: None
dual: False
fit_intercept: True
intercept_scaling: 1
l1_ratio: 0.0
max_iter: 1000
n_jobs: None
penalty: 'deprecated'
random_state: 0
solver: 'lbfgs'
tol: 0.0001
verbose: 0
warm_start: False

Interpretation. Fourteen settings are exposed although only three were supplied; the other eleven are the estimator's defaults. All of them were known before fit() and none was modified by it — a second call after training returns the same dictionary, as section 4.1 verified.

Point of caution on defaults. A model instantiated with no arguments is not a model "without hyperparameters". It is a model in which every hyperparameter took its default value, and those defaults are library conventions, not optima. C=1.0 already regularizes substantially, as section 5.5 makes visible.

Point of caution on versions. This output comes from scikit-learn 1.8, where penalty is being deprecated in favor of l1_ratio, the value 0.0 denoting a pure L2 penalty; earlier releases print penalty: 'l2'. Hyperparameters belong to a library's interface and change with it. Learned parameters belong to the mathematical formulation of the model and do not change name.

5.3 The learned parameters: coef_ and intercept_

print(pd.Series(np.round(lr.coef_[0], 4), index=X.columns).to_string())
print("intercept_:", np.round(lr.intercept_, 4))
print("n_iter_:", lr.n_iter_)
print("classes_:", lr.classes_)

tenure_months   -0.8240
monthly_bill     0.5490
support_calls    0.6133
satisfaction    -0.7968
avg_data_gb     -0.0016
intercept_: [-0.6494]
n_iter_: [6]
classes_: [0 1]

Interpretation. These six numbers — five coefficients and one intercept — are the entirety of what the model learned, and no engineer wrote any of them. The variables having been standardized, the coefficients are comparable with one another: tenure dominates, closely followed by satisfaction, and data usage is negligible.

The last row is the control. avg_data_gb was generated with no link to the target, and the fitted coefficient is −0.0016 — three orders of magnitude below the others. A learned parameter close to zero on a variable known to be uninformative is the expected behavior; a large coefficient on such a variable would have indicated that something in the pipeline was wrong.

Point of caution. n_iter_ and classes_ carry a trailing underscore and exist only after fit(), yet they are not of the same nature as coef_. n_iter_ is a convergence diagnostic: the value 6, far below the budget max_iter=1000, indicates convergence without interruption. A value equal to max_iter would signal a stop on budget exhaustion, hence a model that has not converged and whose coefficients must not be interpreted.

5.4 A tree: imposed hyperparameters against learned structure

from sklearn.tree import DecisionTreeClassifier, export_text

dt = DecisionTreeClassifier(max_depth=3, min_samples_leaf=50, random_state=0)
dt.fit(X, y)

print("nodes:", dt.tree_.node_count, "depth:", dt.get_depth(),
      "leaves:", dt.get_n_leaves())
print(export_text(dt, feature_names=list(X.columns), decimals=2))
print(pd.Series(np.round(dt.feature_importances_, 4), index=X.columns).to_string())

nodes: 15 depth: 3 leaves: 8
|--- tenure_months <= 57.50
|   |--- satisfaction <= 3.50
|   |   |--- monthly_bill <= 65.12
|   |   |   |--- class: 1
|   |   |--- monthly_bill >  65.12
|   |   |   |--- class: 1
|   |--- satisfaction >  3.50
|   |   |--- support_calls <= 1.50
|   |   |   |--- class: 0
|   |   |--- support_calls >  1.50
|   |   |   |--- class: 1
|--- tenure_months >  57.50
|   |--- satisfaction <= 3.50
|   |   |--- support_calls <= 1.50
|   |   |   |--- class: 0
|   |   |--- support_calls >  1.50
|   |   |   |--- class: 0
|   |--- satisfaction >  3.50
|   |   |--- support_calls <= 2.50
|   |   |   |--- class: 0
|   |   |--- support_calls >  2.50
|   |   |   |--- class: 0

tenure_months    0.3965
monthly_bill     0.0437
support_calls    0.1690
satisfaction     0.3908
avg_data_gb      0.0000

What was imposed. Two values only, max_depth=3 and min_samples_leaf=50. The depth obtained is exactly 3: the constraint is binding, and without it the tree would have kept growing.

What was learned. Seven internal splits, each defined by a (variable, threshold) pair, and eight leaves with their class distributions. The values 57.50, 3.50, 65.12, 1.50 and 2.50 all come out of the split search. That support_calls is cut at 1.50 in two branches and at 2.50 in a third is the clearest evidence available that these are locally estimated quantities and not business thresholds: the same variable receives a different cut point depending on which subpopulation the node holds.

The importances. feature_importances_ is a vector summing to 1 that distributes the total impurity reduction across the variables. avg_data_gb gets 0.0000, which here is the correct answer, since the variable carries no signal by construction. In general a zero does not mean "unrelated to the target" but "never selected under these conditions" — a greater depth can hand a nonzero importance to a variable that a shallow tree ignored. The limits of this indicator are covered in chapter 080.

Three of the four leaf pairs predict the same class on both sides. The split was retained anyway because it reduces impurity: the predicted probabilities differ, even where the majority class coincides. A tree read through predict alone hides that, and predict_proba exposes it (chapter 029).

5.5 One hyperparameter determines the number of parameters

for d in [2, 3, 5, 10, None]:
    m = DecisionTreeClassifier(max_depth=d, random_state=0).fit(X, y)
    print(f"max_depth={str(d):>4} -> nodes={m.tree_.node_count:>5}"
          f"  leaves={m.get_n_leaves():>5}  train_acc={m.score(X, y):.4f}")

max_depth=   2 -> nodes=    7  leaves=    4  train_acc=0.7010
max_depth=   3 -> nodes=   15  leaves=    8  train_acc=0.7027
max_depth=   5 -> nodes=   63  leaves=   32  train_acc=0.7535
max_depth=  10 -> nodes=  807  leaves=  404  train_acc=0.8600
max_depth=None -> nodes= 1917  leaves=  959  train_acc=1.0000

Interpretation. A single hyperparameter takes the model from 7 to 1,917 learned nodes, a factor of 274, and training accuracy climbs to 1.0000: the unconstrained tree isolates every observation. That value is not an achievement but memorization. On data where the target was drawn from a Bernoulli variable and the positive rate is 0.38, no honest model classifies its training set perfectly (chapter 031).

The first two rows carry a second lesson, and it is easy to miss. Going from depth 2 to depth 3 doubles the number of learned parameters — 7 nodes to 15 — and buys 0.0017 of training accuracy. The parameter count and the performance are two different quantities, and they do not move together. Section 7 measures what this doubling costs on data the model has not seen.

5.6 The same mechanism on regularization

for C in [0.001, 0.01, 0.1, 1.0, 100.0]:
    m = LogisticRegression(C=C, max_iter=1000).fit(Xs, y)
    print(f"C={C:<8} coef_={np.round(m.coef_[0], 3)}  "
          f"L2_norm={np.linalg.norm(m.coef_):.3f}")

C=0.001    coef_=[-0.309  0.199  0.234 -0.302 -0.002]  L2_norm=0.530
C=0.01     coef_=[-0.677  0.446  0.505 -0.655 -0.002]  L2_norm=1.158
C=0.1      coef_=[-0.806  0.537  0.6   -0.78  -0.002]  L2_norm=1.381
C=1.0      coef_=[-0.824  0.549  0.613 -0.797 -0.002]  L2_norm=1.411
C=100.0    coef_=[-0.826  0.55   0.615 -0.799 -0.002]  L2_norm=1.415

Interpretation. C never appears inside coef_, yet it governs its magnitude: the norm of the coefficient vector rises from 0.530 to 1.415 as C grows from 0.001 to 100. The shrinkage is close to uniform and preserves the ordering of the variables, which is why a heavily regularized model can still rank features correctly while understating every effect. Beyond C=1 the coefficients barely move: the penalty has become negligible against the loss term. Exploring C=10,000 would change nothing here, which is the justification for the bounded logarithmic grids of chapter 035.

Formulation to retain: hyperparameters do not sit inside the model alongside the parameters. They are the constraints under which the parameters were computed.

6. Why hyperparameters are never tuned on the test set

6.1 The principle

Tuning a hyperparameter means comparing several training runs and keeping the best, which requires a set that was not used to fit the parameters. The temptation is to use the test set, since that is the set carrying the performance estimate. That is precisely the use forbidden to it.

Any decision taken while looking at a dataset transfers information from that dataset into the model. The set then stops being unseen, and the score it produces stops estimating performance on new data.

6.2 The quantitative mechanism of selection bias

A score computed on a finite set carries sampling variability. For an accuracy near 0.72 measured on 800 validation observations, the standard error is √(0.72 × 0.28 / 800) ≈ 0.0159. Comparing K configurations and keeping the maximum amounts to selecting, among K noisy draws, the one whose noise was most favorable. The size of that effect is computable.

import numpy as np

rng = np.random.default_rng(0)
sigma = np.sqrt(0.72 * 0.28 / 800)
print(f"Standard error of an accuracy near 0.72 on 800 observations: {sigma:.4f}")
print(f"{'K':>5}  {'E[max of K noise draws]':>24}")
for K in [1, 10, 50, 80, 200]:
    draws = rng.normal(0.0, sigma, size=(200_000, K))
    print(f"{K:>5}  {draws.max(axis=1).mean():>24.4f}")

Standard error of an accuracy near 0.72 on 800 observations: 0.0159
    K   E[max of K noise draws]
    1                    0.0000
   10                    0.0244
   50                    0.0357
   80                    0.0385
  200                    0.0436

Interpretation. With a single candidate the validation score is unbiased. With ten, the winner's score is inflated by about 2.4 accuracy points on average before any real difference between configurations is taken into account. With two hundred, by more than 4 points. Nothing in this calculation depends on the quality of the analysis: it holds for a careful practitioner and a careless one alike, because it is a property of taking a maximum over noisy estimates.

6.3 The effect measured on a real grid

The simulation above says what should happen. The experiment below checks that it does, on the chapter's dataset, with a grid of 80 tree configurations.

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X_pool, X_test, y_pool, y_test = train_test_split(
    X, y, test_size=800, stratify=y, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(
    X_pool, y_pool, test_size=800, stratify=y_pool, random_state=0)
print("train:", X_train.shape[0], "validation:", X_val.shape[0],
      "test:", X_test.shape[0])

grid = [{"max_depth": d, "min_samples_leaf": m, "criterion": c}
        for d in [2, 3, 4, 5, 6, 8, 10, 14]
        for m in [1, 5, 20, 50, 100]
        for c in ["gini", "entropy"]]
print("configurations compared:", len(grid))

rows = []
for cfg in grid:
    m = DecisionTreeClassifier(random_state=0, **cfg).fit(X_train, y_train)
    rows.append((cfg, m.score(X_val, y_val), m.score(X_test, y_test)))

best_cfg, best_val, best_test = max(rows, key=lambda r: r[1])
val_scores = np.array([r[1] for r in rows])
print("best configuration :", best_cfg)
print(f"validation accuracy of the winner: {best_val:.4f}")
print(f"test accuracy of the same model  : {best_test:.4f}")
print(f"selection optimism               : {best_val - best_test:+.4f}")
print(f"mean validation accuracy over the grid: {val_scores.mean():.4f}")
print(f"standard deviation over the grid      : {val_scores.std():.4f}")

train: 2400 validation: 800 test: 800
configurations compared: 80
best configuration : {'max_depth': 4, 'min_samples_leaf': 100, 'criterion': 'gini'}
validation accuracy of the winner: 0.7212
test accuracy of the same model  : 0.6887
selection optimism               : +0.0325
mean validation accuracy over the grid: 0.7043
standard deviation over the grid      : 0.0151

Interpretation. The winning configuration scores 0.7212 on validation and 0.6887 on test — the same model, the same parameters, two different sets. The gap of 3.3 accuracy points is not a defect of the model. It is the price of having looked at the validation set 80 times, and the simulation of section 6.2 predicted 0.0385 for K = 80 against 0.0325 observed. Had the validation score been reported as the deliverable's performance, the system would have been oversold by three points before it ever reached production.

Two further readings of the same output. The spread of validation scores across the grid, 0.0151, is almost exactly the sampling standard error computed in section 6.2, which says that the 80 configurations are largely indistinguishable in truth and that the ranking between them is mostly noise. And the winner, max_depth=4 with min_samples_leaf=100, is a heavily constrained tree — the grid did contain deeper, freer configurations, and validation rejected them. The selection did its job; only its score cannot be believed.

6.4 Operational formulations

• The test set is a single-use budget: it is consumed when you look at it, not when you modify it.
• The moment any choice is arbitrated on a set — a hyperparameter, an algorithm, a threshold, a subset of variables, an imputation strategy — that set is acting as a validation set, whatever the variable holding it is called. A test set consulted repeatedly degrades into a validation set, gradually and with no signal in the metrics.
• The gap between the performance announced in a meeting and the performance observed in production has this as one of its two principal causes. The other is leakage, covered in chapter 028.

Cross-references: the construction of the three sets is covered in chapter 026, the special splitting schemes in chapter 027, cross-validation in chapter 034, hyperparameter search procedures in chapter 035, and the choice of the decision threshold in chapter 062.

7. Model capacity and the bias-variance tradeoff

Sections 5 and 6 established two facts separately. A hyperparameter governs how many parameters exist and how large they are allowed to be. A score measured on a set used for decisions is optimistic. Capacity is the concept that joins them: it names what a hyperparameter actually controls, and it explains why more of it stops helping.

7.1 Capacity, defined

How to read the diagram. Two arrows leave H. Which one is realized is not a property of H alone: it depends on how much data there is and how noisy it is. The same max_depth=10 underfits a million clean rows and overfits three hundred noisy ones. This is why no hyperparameter value transfers between projects.

7.2 Direction of effect, hyperparameter by hyperparameter

The last three rows are where the table has to be read carefully rather than memorized, and section 7.6 measures the divergence between the two n_estimators.

7.3 max_depth against the train-test gap, measured

The mechanism is not argued here, it is executed. The dataset is the 4,000 customers of section 5.1, split once into 3,000 training and 1,000 test observations. Only max_depth varies.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

df = pd.read_csv("telecom_churn_4k.csv")
y = df["churned"]
X = df.drop(columns="churned")

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=1000, stratify=y, random_state=0)

print(f"{'max_depth':>9} {'nodes':>6} {'train':>7} {'test':>7} {'gap':>7}")
for d in [1, 2, 3, 4, 6, 8, 12, 20, None]:
    m = DecisionTreeClassifier(max_depth=d, random_state=0).fit(X_train, y_train)
    tr = m.score(X_train, y_train)
    te = m.score(X_test, y_test)
    print(f"{str(d):>9} {m.tree_.node_count:>6} {tr:>7.4f} {te:>7.4f} {tr - te:>+7.4f}")

max_depth  nodes   train    test     gap
        1      3  0.6493  0.6230 +0.0263
        2      7  0.6993  0.7080 -0.0087
        3     15  0.7083  0.7170 -0.0087
        4     31  0.7357  0.7080 +0.0277
        6    117  0.7717  0.7240 +0.0477
        8    333  0.8263  0.7080 +0.1183
       12    941  0.9217  0.6910 +0.2307
       20   1491  0.9963  0.6820 +0.3143
     None   1517  1.0000  0.6680 +0.3320

Upper curve: training accuracy. Lower curve: test accuracy. Plotted from the output above.

Interpretation. Three regimes appear in one table, and the boundaries between them are not where a first reading expects.

At max_depth=1 the model holds three nodes and fails on both sets: 0.6493 on training, 0.6230 on test. A single split cannot represent a phenomenon driven by four variables. This is underfitting, and its signature is that both scores are poor and close together. The gap of +0.0263 is not a warning sign here; the level is.

Between depth 2 and depth 6 the two curves rise together. Test accuracy peaks at 0.7240 for max_depth=6, with 117 learned nodes. Note that depths 2 and 3 show a negative gap: test accuracy exceeds training accuracy by 0.0087. That is not an anomaly requiring explanation, it is sampling noise on 1,000 test observations, whose standard error is about 0.014. A gap smaller than the standard error of the scores means nothing.

From depth 8 onward the curves separate for good. Training accuracy climbs to 1.0000 while test accuracy falls to 0.6680 — below the max_depth=2 model, which learned 7 nodes instead of 1,517. The unconstrained tree is worse on new data than a tree two hundred times smaller, and it is worse while looking perfect on the data it was fitted to. This is the entire argument for holding out data, made in one row of a table.

How to read the diagram. The three states carry the measured figures of the sweep above, so the transitions are not schematic. The arrow back from overfitting to a well-fitted model is the one that matters operationally: it is always available, and it is always cheaper than collecting more data.

7.4 C against the fitted parameters, measured

The tree makes capacity visible as a count of nodes. A linear model makes it visible as the magnitude of the coefficients. To expose the regimes on a linear model, two things are needed: a hypothesis space rich enough to overfit — here a degree-3 polynomial expansion of the five variables, 55 derived features — and a training set small enough for the noise to matter, here 300 observations.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

X_pool, X_test, y_pool, y_test = train_test_split(
    X, y, test_size=1000, stratify=y, random_state=0)
X_train, _, y_train, _ = train_test_split(
    X_pool, y_pool, train_size=300, stratify=y_pool, random_state=0)

print(f"{'C':>9} {'|coef|':>8} {'train':>7} {'test':>7} {'gap':>7}")
for C in [1e-4, 1e-3, 1e-2, 1e-1, 1.0, 1e2, 1e4]:
    p = make_pipeline(PolynomialFeatures(degree=3, include_bias=False),
                      StandardScaler(),
                      LogisticRegression(C=C, max_iter=20_000, random_state=0))
    p.fit(X_train, y_train)
    coef = p[-1].coef_[0]
    tr = p.score(X_train, y_train)
    te = p.score(X_test, y_test)
    print(f"{C:>9.4g} {np.linalg.norm(coef):>8.3f} {tr:>7.4f} {te:>7.4f} {tr - te:>+7.4f}")

        C   |coef|   train    test     gap
   0.0001    0.022  0.6167  0.6160 +0.0007
    0.001    0.143  0.6733  0.6820 -0.0087
     0.01    0.416  0.7567  0.7320 +0.0247
      0.1    0.814  0.7500  0.7270 +0.0230
        1    2.169  0.7767  0.7110 +0.0657
      100   20.748  0.8133  0.6850 +0.1283
    1e+04   43.908  0.8100  0.6750 +0.1350

Interpretation. The same three regimes appear, driven by a hyperparameter that changes no structure at all. The 55 features are present in every row; what changes is how large their coefficients are permitted to become.

At C=0.0001 the penalty crushes the coefficient vector to a norm of 0.022. The model is effectively constant: 0.6167 on training and 0.6160 on test, against a majority-class baseline of 0.6160 measured in section 8.5. Strong regularization has removed the capacity to represent anything at all — underfitting produced not by a shortage of features but by a refusal to use them.

Test accuracy peaks at 0.7320 for C=0.01, where the coefficient norm is 0.416. Past that point the norm grows by two orders of magnitude, to 43.908, while training accuracy gains 0.0533 and test accuracy loses 0.0570. The last two rows are the clearest statement of what excessive capacity buys: a coefficient vector one hundred times larger, describing the 300 training observations slightly better and new customers substantially worse.

Point of caution on transferability. Nothing in this table says C=0.01 is a good value. It is the best of seven values, on 300 observations, with a degree-3 expansion, on this dataset. Section 5.6 ran the same sweep on 4,000 observations without the expansion and found the coefficients frozen beyond C=1. The optimum moved because the sample size and the hypothesis space moved. That is the general case, not an exception.

7.5 Variance made visible: the same configuration, forty samples

Bias and variance are usually defined as an expectation over training sets, which makes them sound unmeasurable in practice. They are not. Refitting the same configuration on repeated subsamples and watching how much the predictions move measures the variance component directly.

rng = np.random.default_rng(7)
R = 40

print(f"{'max_depth':>9} {'train acc':>10} {'test acc':>9} {'disagreement':>13}")
for d in [2, 4, 8, None]:
    preds, tr = [], []
    for _ in range(R):
        idx = rng.choice(len(X_pool), size=1500, replace=False)
        Xi, yi = X_pool.iloc[idx], y_pool.iloc[idx]
        m = DecisionTreeClassifier(max_depth=d, random_state=0).fit(Xi, yi)
        preds.append(m.predict(X_test))
        tr.append(m.score(Xi, yi))
    P = np.array(preds)
    p1 = P.mean(axis=0)
    disagreement = (2 * p1 * (1 - p1)).mean()
    accuracy = (P == y_test.to_numpy()).mean()
    print(f"{str(d):>9} {np.mean(tr):>10.4f} {accuracy:>9.4f} {disagreement:>13.4f}")

max_depth  train acc  test acc  disagreement
        2     0.7007    0.7032        0.0743
        4     0.7496    0.7089        0.1508
        8     0.8548    0.6832        0.2379
     None     1.0000    0.6460        0.3234

How the disagreement column is built. For each test customer, the forty refitted trees vote. If a fraction p of them predict the positive class, the probability that two runs drawn at random disagree on that customer is 2p(1 − p). Averaging over the test set gives a single number between 0 and 0.5 that measures instability, with 0 meaning every refit agrees everywhere.

Interpretation. The two proxies move in opposite directions, exactly as the decomposition predicts. Failure to fit the training data — one minus training accuracy, a proxy for bias — falls from 0.2993 at depth 2 to 0.0000 for the unconstrained tree. Instability across resamples — a proxy for variance — rises from 0.0743 to 0.3234 over the same range. Test accuracy, which pays for both, peaks in between at depth 4.

The unconstrained tree deserves its own sentence. It fits every training set it is given perfectly, and yet two such trees, fitted on two overlapping samples from the same population, disagree on about a third of new customers. Nothing about the phenomenon changed between the two runs; only the sample did. A model whose predictions depend that heavily on which 1,500 rows it happened to see is not describing customers, it is describing that sample.

Coordinates are the two measured proxies of the table above, each rescaled to the unit square: the horizontal axis is the disagreement rate, the vertical axis is training accuracy. No configuration in the sweep reaches the target region, which is the ordinary situation: a single decision tree trades one for the other along a line. Escaping that line requires a different algorithm — averaging many trees, which is what chapter 038 is about.

7.6 One name, two opposite behaviors: n_estimators

The capacity table gives n_estimators two different rows because the same argument name does two different things. In a random forest, trees are fitted independently on bootstrap samples and averaged: adding trees reduces the variance of the average and cannot increase capacity. In gradient boosting, each tree is fitted to the residual errors of the ensemble so far: adding trees adds corrective terms, which is a direct increase in capacity.

from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=1000, stratify=y, random_state=0)

header = (f"{'n_estimators':>12} | {'RF train':>8} {'RF test':>8} {'RF gap':>7}"
          f" | {'GB train':>8} {'GB test':>8} {'GB gap':>7}")
print(header)
for n in [5, 25, 100, 400, 1000]:
    rf = RandomForestClassifier(n_estimators=n, max_depth=6,
                                random_state=0, n_jobs=-1).fit(X_train, y_train)
    gb = GradientBoostingClassifier(n_estimators=n, max_depth=3,
                                    learning_rate=0.1,
                                    random_state=0).fit(X_train, y_train)
    rt, rs = rf.score(X_train, y_train), rf.score(X_test, y_test)
    gt, gs = gb.score(X_train, y_train), gb.score(X_test, y_test)
    print(f"{n:>12} | {rt:>8.4f} {rs:>8.4f} {rt - rs:>+7.4f}"
          f" | {gt:>8.4f} {gs:>8.4f} {gt - gs:>+7.4f}")

n_estimators | RF train  RF test  RF gap | GB train  GB test  GB gap
           5 |   0.7813   0.7090 +0.0723 |   0.6893   0.6790 +0.0103
          25 |   0.7893   0.7240 +0.0653 |   0.7680   0.7280 +0.0400
         100 |   0.7897   0.7270 +0.0627 |   0.7970   0.7210 +0.0760
         400 |   0.7893   0.7310 +0.0583 |   0.8773   0.7250 +0.1523
        1000 |   0.7897   0.7300 +0.0597 |   0.9467   0.7060 +0.2407

Interpretation. The two halves of the table behave nothing alike.

The forest converges. Training accuracy is flat at 0.789 from 25 trees onward, test accuracy drifts from 0.7090 to 0.7310, and the gap narrows from +0.0723 to +0.0597. Going from 400 to 1,000 trees changes test accuracy by 0.0010 and multiplies the fitting cost by 2.5. There is a point past which more trees buy compute time and nothing else, and it arrives early.

The boosting model diverges. Training accuracy climbs from 0.6893 to 0.9467, test accuracy peaks at 0.7280 for 25 iterations and then falls to 0.7060, and the gap grows from +0.0103 to +0.2407 — a twenty-three-fold increase. The configuration that memorizes the training set best is the worst of the five on new data.

The lesson generalizes past this pair. A hyperparameter's name does not determine its effect; the algorithm's mechanism does. Reading the direction of an effect off the argument name is how a practitioner ends up tuning a forest with an early stopping mindset, or a boosting model with a "more is always safe" one. Bagging is covered in chapter 038, boosting in chapter 039.

7.7 The tradeoff, stated

Insufficient capacity produces a systematic error, the bias: the model fails on the training set and on the test set alike, and the two failures are of similar size. Excessive capacity produces sensitivity to the particular training sample, the variance: the model succeeds on the training set and fails on new data, and the distance between the two is the diagnostic.

Tuning capacity hyperparameters means arbitrating between these two regimes. That arbitration is the subject of chapter 032; overfitting and underfitting as observable phenomena are chapter 031; the corrective levers — regularization, pruning, more data, feature reduction, ensembling — are chapter 033.

8. The connected vocabulary: loss, metric, score, threshold, naive baseline

These five notions are introduced here because they cannot be separated from the distinction just made. The loss is what the optimization minimizes in order to produce the parameters. The metric is what validation compares in order to choose the hyperparameters. The threshold and the baseline are two engineering decisions that nothing learns. Each is treated in depth later; what follows fixes the definitions and the boundaries between them.

How to read the diagram. The chain has exactly one link that training produces, and it is the parameters. The loss is chosen before, the threshold is chosen after, the metric is chosen outside, and the baseline never touches the model at all. Everything else in the picture is an engineering decision.

8.1 The loss function

Point of caution on configurability. The loss is largely imposed by the algorithm, but not always. SGDClassifier exposes loss directly, taking hinge, log_loss or modified_huber; HistGradientBoostingRegressor exposes loss with squared_error, absolute_error and quantile options; a decision tree exposes criterion. Where such an argument exists, it is a hyperparameter like any other, and one of the few whose effect changes what the model is rather than how tightly it fits.

8.2 The evaluation metric

The last row is the one worth carrying away. Changing the metric after training changes nothing inside the model. It changes which model you would have selected, which is why the metric must be chosen before the comparison starts and not after the results arrive.

A worked case where the two diverge. One fitted model, one loss, six metrics, all computed on the same 1,000 test predictions.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import (accuracy_score, f1_score, log_loss,
                             precision_score, recall_score, roc_auc_score)

pipe = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=1000, random_state=0)).fit(X_train, y_train)
proba = pipe.predict_proba(X_test)[:, 1]

print(f"log loss (the training loss) : {log_loss(y_test, proba):.4f}")
print(f"accuracy  at threshold 0.50  : {accuracy_score(y_test, proba >= 0.50):.4f}")
print(f"precision at threshold 0.50  : {precision_score(y_test, proba >= 0.50):.4f}")
print(f"recall    at threshold 0.50  : {recall_score(y_test, proba >= 0.50):.4f}")
print(f"F1        at threshold 0.50  : {f1_score(y_test, proba >= 0.50):.4f}")
print(f"ROC AUC   (threshold-free)   : {roc_auc_score(y_test, proba):.4f}")

log loss (the training loss) : 0.4960
accuracy  at threshold 0.50  : 0.7290
precision at threshold 0.50  : 0.6760
recall    at threshold 0.50  : 0.5651
F1        at threshold 0.50  : 0.6156
ROC AUC   (threshold-free)   : 0.8242

Interpretation. Six numbers describe one model. Announcing "the model scores 0.82" and announcing "the model scores 0.57" are both true statements about this same object, and they support opposite decisions. The AUC of 0.8242 says the ranking of customers by churn risk is good. The recall of 0.5651 says that at the default threshold, the model misses 43% of the customers who will actually leave. A retention campaign is designed against the second number, not the first.

Notice also that the loss and the metrics disagree about what "good" means: cross-entropy rewards well-calibrated probabilities everywhere, while recall cares only about which side of a cutoff each observation falls on. Chapter 061 develops that gap.

8.3 Score

8.4 The decision threshold

The threshold changes the decisions without touching a single learned parameter. The model below is fitted once; only the cutoff moves.

print(f"{'threshold':>9} {'predicted positives':>20} {'precision':>10}"
      f" {'recall':>8} {'F1':>8} {'accuracy':>9}")
for t in [0.20, 0.30, 0.40, 0.50, 0.60, 0.70]:
    pred = (proba >= t).astype(int)
    print(f"{t:>9.2f} {pred.sum():>20} "
          f"{precision_score(y_test, pred, zero_division=0):>10.4f} "
          f"{recall_score(y_test, pred):>8.4f} {f1_score(y_test, pred):>8.4f} "
          f"{accuracy_score(y_test, pred):>9.4f}")

threshold  predicted positives  precision   recall       F1  accuracy
     0.20                  689     0.5196   0.9323   0.6673    0.6430
     0.30                  535     0.6000   0.8359   0.6986    0.7230
     0.40                  418     0.6603   0.7188   0.6883    0.7500
     0.50                  321     0.6760   0.5651   0.6156    0.7290
     0.60                  212     0.7689   0.4245   0.5470    0.7300
     0.70                  131     0.8015   0.2734   0.4078    0.6950

Rising curve: precision. Falling curve: recall. Plotted from the output above. The two curves cross between 0.40 and 0.50, which is where F1 is maximized.

Interpretation. Six rows, one model, six different systems. At a threshold of 0.20 the model flags 689 customers out of 1,000 and catches 93% of the churners, at the cost of being wrong about half the time it raises a flag. At 0.70 it flags 131 customers, is right 80% of the time, and misses nearly three churners out of four.

None of these rows is the correct one in the abstract. If a retention offer costs $40 and a lost customer costs $600, the arithmetic decides: at threshold 0.20 the campaign costs 689 × $40 = $27,560 and saves 358 churners; at threshold 0.70 it costs 131 × $40 = $5,240 and saves 105. The threshold is not a modeling parameter, it is a budget decision expressed in probability units. Chapter 062 carries out the full optimization, and lab 088 does it on a real cost matrix.

Point of caution. Notice that accuracy peaks at 0.40, not at the default 0.50. The default cutoff is not even optimal for the metric it is most often reported with. predict() applies 0.50 because a library has to apply something.

8.5 The naive baseline

for strategy in ["most_frequent", "stratified", "prior"]:
    d = DummyClassifier(strategy=strategy, random_state=0).fit(X_train, y_train)
    dp = d.predict_proba(X_test)[:, 1]
    print(f"{strategy:>14}: accuracy={d.score(X_test, y_test):.4f}"
          f"  log_loss={log_loss(y_test, dp):.4f}")
print(f"{'logistic':>14}: accuracy={pipe.score(X_test, y_test):.4f}"
      f"  log_loss={log_loss(y_test, proba):.4f}")

 most_frequent: accuracy=0.6160  log_loss=13.8408
    stratified: accuracy=0.5170  log_loss=17.4091
         prior: accuracy=0.6160  log_loss=0.6660
      logistic: accuracy=0.7290  log_loss=0.4960

Interpretation. The fitted model reaches 0.7290 accuracy. Reported alone, that number sounds like a working system. Against the baseline it is worth exactly 0.7290 − 0.6160 = 0.1130 of accuracy, or eleven correctly classified customers per hundred beyond what predicting "nobody churns" achieves. Both statements are true; only the second one is informative.

The log loss column carries a separate lesson. most_frequent scores 13.8408 and stratified scores 17.4091 — enormous values, because both strategies emit hard 0 or 1 probabilities and log loss punishes a confident error without mercy, up to the clipping bound scikit-learn applies to avoid an infinity. The prior strategy emits the class frequency 0.384 for every customer, never commits, and scores 0.6660. Three baselines, the same accuracy for two of them, log losses spread over a factor of twenty-six. The choice of baseline is itself a choice of what "trivial" means, and it has to be made with the metric in view.

8.6 The full chain, in order of operations

The five notions are not simultaneous. They enter at specific moments, and the order is what makes the protocol of section 6 enforceable rather than a matter of discipline.

How to read the diagram. Only step 4 returns learned quantities. Every other arrow leaving the engineer is a decision, and every decision taken against the validation set is a reason the validation score cannot be reported as the system's performance. The single arrow to the test set is the deliverable, and it fires once.

Reference sheet

The two categories

PARAMETER        learned by the optimization procedure, during fit()
                 coefficients, intercept, weights and biases,
                 split variables and thresholds, tree topology,
                 dual coefficients and support vectors,
                 preprocessor statistics (mean_, scale_, statistics_)

HYPERPARAMETER   fixed by the engineer, before fit()
                 max_depth, min_samples_leaf, n_estimators, k,
                 learning_rate, alpha, C, gamma, kernel,
                 hidden_layer_sizes, class_weight, random_state,
                 and the decision threshold

The single test

Is this quantity adjusted by the optimization procedure executed during fit(), or did it have to be known before that procedure could start?

If it had to be known in advance, it is a hyperparameter. If it came out of the run and enters a prediction, it is a parameter. If it came out of the run and enters no prediction, it is an execution result.

The regularized objective

J(θ) = data_loss(θ) + λ · penalty(θ)

Regularization requires comparable scales. Always standardize inside a pipeline (chapters 023 and 076).

Parameter counts

Reading table for a train-test gap

A gap smaller than the standard error of the scores is not a gap. For accuracy on n test observations, that standard error is √(p(1−p)/n); on 1,000 observations near 0.72 it is about 0.014.

Conditions of use and non-use

The exact scikit-learn calls

# Read the hyperparameters, before or after fit; the answer is the same.
model.get_params()
model.get_params(deep=True)          # inside a Pipeline: step__argument
model.set_params(max_depth=6)        # invalidates the fit

# Read the learned quantities; these exist only after fit.
model.coef_, model.intercept_        # linear models
model.feature_importances_           # trees and ensembles
model.tree_.node_count               # tree structure
model.support_vectors_, model.dual_coef_, model.n_support_   # SVM
model.coefs_, model.intercepts_      # MLP, one array per transition
scaler.mean_, scaler.scale_          # StandardScaler
model.classes_, model.n_features_in_ # metadata, learned but not predictive
model.n_iter_                        # execution diagnostic

# The convention test.
from sklearn.exceptions import NotFittedError
try:
    model.coef_
except NotFittedError:
    print("not fitted: every trailing-underscore attribute is learned")

# Capacity, measured rather than assumed.
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
gap = train_score - test_score

# The baseline, before any comparison.
from sklearn.dummy import DummyClassifier
DummyClassifier(strategy="most_frequent").fit(X_train, y_train).score(X_test, y_test)

9. Common reasoning mistakes

10. Summary

THE TWO CATEGORIES
    PARAMETER       learned by the optimization procedure, during fit()
                    coefficients, weights and biases, tree thresholds and
                    structure, dual coefficients and support vectors
    HYPERPARAMETER  fixed by the engineer, before fit()
                    max_depth, n_estimators, k, learning_rate, alpha, C

THE SINGLE TEST OF DISTINCTION
    Is this quantity adjusted by the optimization procedure,
    or did it have to be known for that procedure to start?

THE SCIKIT-LEARN CONVENTION
    get_params()               -> the hyperparameters
    attribute_ (underscore)    -> the learned quantities
    coef_, intercept_, feature_importances_, tree_, support_vectors_
    Before fit(), these attributes do not exist: NotFittedError.

THE REFERENCE ANALOGY
    recipe                 = algorithm
    oven settings          = hyperparameters
    what the batter becomes = learned parameters
    cake                   = model

WHAT HYPERPARAMETERS GOVERN
    capacity              max_depth, C, alpha, k, hidden_layer_sizes
    optimization          learning_rate, solver, max_iter, tol
    ensemble structure    n_estimators, max_features, subsample
    problem handling      class_weight, criterion, decision threshold
    execution             n_jobs, verbose, random_state

CAPACITY
    Hyperparameters delimit the hypothesis space H.
    Parameters select one element h inside H.
    Too little capacity -> high bias, underfitting.
    Too much capacity   -> high variance, overfitting.
    Measured on this chapter's data:
        max_depth = 1     train 0.6493  test 0.6230  gap +0.0263
        max_depth = 6     train 0.7717  test 0.7240  gap +0.0477
        max_depth = none  train 1.0000  test 0.6680  gap +0.3320

PROTOCOL RULE
    Training   -> fits the parameters
    Validation -> chooses the hyperparameters and the threshold
    Test       -> estimates performance, consulted once
    A set you decide on stops being an evaluation set.
    Measured cost of 80 comparisons: validation 0.7212, test 0.6887.

THE CONNECTED VOCABULARY
    loss       what the optimization minimizes to produce the parameters
    metric     what you compare to choose the hyperparameters
    score      the model's continuous output, or the value of a metric
    threshold  the cutoff applied to a score; 0.50 is a convention, not learned
    baseline   the bar below which the model adds nothing

Summary statement

A trained model reads on two levels. The hyperparameters, fixed before learning, delimit the family of accessible functions and the way the search through it will be conducted. The parameters, produced by that search, name the function finally retained. The first are tuned by comparing successive training runs on a validation set, never on the test set, whose single consultation is the only unbiased estimate of expected performance available.

Associated quizzes

• 009.1-quiz-learned-parameter.md
• 009.2-quiz-hyperparameter.md
• 009.3-quiz-distinction-criterion.md
• 009.4-quiz-scikit-learn-inspection.md
• 009.5-quiz-tuning-and-the-test-set.md
• 009.6-quiz-capacity-bias-variance.md
• 009.7-quiz-loss-metric-score-threshold-baseline.md

Next chapter : 010.0-classification-or-regression.md