The four statements below are the kind of sentence that appears routinely in project reports and stand-up notes. Three of them are wrong.
| Statement | Status | Reason |
|---|---|---|
| "We trained a random forest on 40,000 observations." | Acceptable | Accepted ellipsis: trained with the random forest algorithm |
| "The algorithm predicts a churn probability of 0.72." | Incorrect | An algorithm does not predict; the model produces the prediction |
| "We deployed an XGBoost in production." | Incorrect | What is deployed is a model produced by XGBoost, not the library |
| "The algorithm learned that longer tenure reduces churn risk." | Incorrect | The algorithm runs a procedure; what was learned lives in the model's parameters |
The gap is not merely lexical. What gets versioned, what is reproducible, what must be re-executed when the data changes, and what is handed over to operations are different objects depending on whether the sentence is about the algorithm or about the model. Section 3 makes that consequence explicit.
A useful test, applied throughout this chapter: substitute the phrase "the procedure" for the term under discussion. If the sentence still holds, the term denotes the algorithm. Substitute "the trained file" instead; if that reads correctly, the term denotes the model.
Rigorous definition. A finite, unambiguous sequence of elementary operations that, from inputs belonging to a specified domain, produces an output in a finite number of steps. The classical characterization lists five properties (Knuth, The Art of Computer Programming, 1968): finiteness, unambiguous definition of each step, specified inputs, specified outputs, and effectiveness of the operations.
In plain terms. A method written once and for all, describing what to do and in what order, independently of the values it will later be run on.
Point of caution. An algorithm is a description of a procedure. It holds no state, contains no knowledge about any particular domain, and does not change when the data changes.
Rigorous definition. A special case of the above: a procedure that, applied to a dataset D = {(x₁, y₁), …, (xₙ, yₙ)} and to a hyperparameter configuration λ, selects an element ĥ of a hypothesis space H according to an explicit criterion evaluated on D. Formally, A : (D, λ) ⟼ ĥ ∈ H.
In plain terms. The method that turns a table of already labeled examples into a decision rule. It describes how to search, not what will be found. Three elements must be named for a learning algorithm to be fully specified: the hypothesis space it explores (section 1.3), the criterion evaluated on the data, and the search strategy used to explore that space (section 4.2).
Point of caution. The algorithm is identical across every project that uses
it. RandomForestClassifier is the same object for every user of a given
scikit-learn release. What differs from one project to the next is the model
produced, never the algorithm.
Rigorous definition. The set H of all candidate functions h : X → Y that the
algorithm is structurally capable of producing, where X is the space of
explanatory variables and Y the space of the target. H is determined jointly by
the family of functions chosen and by the hyperparameters that constrain its
shape: setting max_depth=3 does not modify the algorithm, but it restricts H to
trees of depth at most 3.
In plain terms. The set of all rules the method is allowed to build. It will never produce a rule outside that set, no matter how much data it is given. For a linear regression on two variables, H is the set of functions h(x) = β₀ + β₁x₁ + β₂x₂ — infinite in cardinality, but parameterized by three real numbers. No dataset whatsoever will make it produce a step function or a periodic one.
Point of caution. An underperforming model underperforms for one of two distinct reasons: the right function does not belong to H (approximation error, tied to the choice of algorithm), or it belongs to H but was not found (estimation error, tied to the data and to the optimization). Chapter 032 develops this decomposition.
Rigorous definition (Mitchell, 1980). The set of additional assumptions, not deducible from the training data, that an algorithm relies on in order to prefer certain generalizations over others among those consistent with the observed data. It takes two forms: restriction of the hypothesis space, and preference within H — for instance, preferring the simplest hypothesis at equal performance.
In plain terms. A set of examples never determines a single rule; infinitely many rules pass through the same points. The inductive bias is the set of preferences built into the method that decide between them.
Related result (Wolpert, 1996 — No Free Lunch). Averaged over the set of all possible problems, no algorithm outperforms any other. Its performance on a given problem comes from the fit between its inductive bias and the actual structure of the phenomenon. Choosing an algorithm therefore amounts to placing a bet on that structure (chapter 049).
Point of caution. Learning without an inductive bias is impossible. An algorithm with no preference at all could assert nothing about an observation absent from the training set.
The two notions lock together. The hypothesis space says which rules exist as candidates; the inductive bias says which of them the procedure will favor when several fit the data equally well. Every practical choice you make about an algorithm — the family, the depth limit, the penalty term — acts on one of these two levers and on nothing else.
| Algorithm | Hypothesis space H | Form of a hypothesis | Main inductive bias |
|---|---|---|---|
| Linear regression | Affine functions of X | Weighted sum of the variables | Effects are linear and additive |
| Logistic regression | Logistic transforms of affine functions | Sigmoid of a linear combination | The decision boundary is linear |
| Decision tree (CART) | Partitions of X by axis-aligned splits | A sequence of threshold tests, constant on each leaf | Axis-parallel splits, preference for short trees |
| k-nearest neighbors | Locally constant functions | Local vote or local average | Local continuity: nearby observations have nearby targets |
| Naive Bayes | Factorized conditional distributions | Product of per-variable likelihoods | Variables are conditionally independent given the class |
| Multilayer perceptron | Compositions of affine maps and nonlinearities | A layered computation graph | Compositionality and smoothness of the representable functions |
How to read this table. Each row states a different bet about the structure of the phenomenon. A sharp threshold effect is served badly by a linear regression and well by a tree; a smooth additive phenomenon, the reverse. None of the rows is a statement about accuracy — accuracy depends on whether the bet matches the data at hand.
Rigorous definition. The element ĥ ∈ H selected by the algorithm A at the end of its execution on a dataset D with hyperparameters λ. A model is fully specified by two components: a structure — the functional form, fixed by A and λ — and a set of parameter values fitted from D.
Operational definition. A software artifact implementing a deterministic function from the explanatory variables to the output space, whose behavior depends entirely on the dataset it was fitted on, and which can be serialized, versioned, transported and evaluated independently of the procedure that produced it.
In plain terms. The result of training: a frozen decision rule, in memory or in a file, that answers a precise question for a given observation.
Point of caution. The model is not the file. The file is one serialization
medium for it, in the same way that a score is not the music. The same model
exports to joblib, to ONNX or to PMML without changing nature.
| Model family | Structure fixed by the algorithm and λ | Parameters fitted on the data | Order of magnitude |
|---|---|---|---|
| Linear regression on p variables | One affine equation | p coefficients and one intercept | p + 1 |
| Logistic regression on p variables | One affine equation composed with a logistic | p coefficients and one intercept | p + 1 |
| Decision tree | A binary tree of bounded depth | Split variable and threshold at each node, value at each leaf | 10¹ to 10³ |
| Random forest of 200 trees | 200 trees aggregated by vote or average | All parameters of the 200 trees | 10⁴ to 10⁷ |
| k-nearest neighbors | A distance metric and an integer k | None in the strict sense: the training set is memorized | n × p stored values |
| Multilayer perceptron | Number of layers, widths, activations | Weights and biases of every connection | 10³ to 10¹¹ |
Point of caution — k-nearest neighbors. This algorithm produces no parameters in the usual sense. It retains the training observations and defers all computation to prediction time, which is why it is called a lazy learner. The definition of the model as an artifact still holds: the structure is the voting rule over the k neighbors, and the fitted content is the memorized set (chapter 040).
The boundary between what a human sets and what the procedure learns is the subject of chapter 009. The principle needed here is narrower: hyperparameters define H, and parameters designate the element retained inside H.
In scikit-learn, attributes whose name ends with a trailing underscore exist only after training. Their presence is the operational criterion that separates an instantiated algorithm from a model.
import pandas as pd
from sklearn.linear_model import LinearRegression
FEATURES = ["area_sqm", "room_count", "age_years"]
north = pd.read_csv("market_north.csv")
X, y = north[FEATURES], north["price"]
def learned(obj):
return [a for a in dir(obj) if a.endswith("_") and not a.startswith("_")]
estimator = LinearRegression()
print("Before fit - hyperparameters :", estimator.get_params())
print("Before fit - learned attributes:", learned(estimator))
estimator.fit(X, y)
print("After fit - learned attributes:", learned(estimator))Before fit - hyperparameters : {'copy_X': True, 'fit_intercept': True,
'n_jobs': None, 'positive': False, 'tol': 1e-06}
Before fit - learned attributes: []
After fit - learned attributes: ['coef_', 'feature_names_in_', 'intercept_',
'n_features_in_', 'rank_', 'singular_']Interpretation. Before the call, the object materializes the algorithm and the
hypothesis space: its hyperparameters are set, its knowledge is empty. After the
call, coef_ and intercept_ carry the information induced from the data, while
n_features_in_ and feature_names_in_ record the input schema the model now
expects (section 5.2). One and the same object plays two roles in succession —
scikit-learn calls it an estimator both before and after fit, which is exactly
why the vocabulary needs the discipline described in this chapter.
| Criterion | Learning algorithm | Model |
|---|---|---|
| Nature | A procedure | An artifact |
| Moment of existence | Before any data | After training |
| Dependence on data | None | Total |
| Content | Instructions | A structure and parameter values |
| Determined by | A human design choice | The training data |
| Physical support | Code, an installed library | An object in memory, a serialized file |
| Uniqueness | One algorithm, many models | One model per dated training run |
| What is versioned | A software dependency (scikit-learn==1.8.0) | An identified deliverable (churn_model_v3.joblib) |
| Is it deployed | No, except to retrain | Yes |
| How it ages | By obsolescence of the library | By drift of the data distribution (chapter 082) |
A recipe is a text. It cannot be eaten, it does not spoil, it stays identical however many times it is executed, and it photocopies without loss. A cake is an object. It results from executing the recipe on specific ingredients, in an oven set a certain way, on a given day. Two cakes from the same recipe differ if the ingredients differ; a cake can be transported, dated, and will eventually spoil.
| Culinary element | Counterpart in supervised learning | Justification |
|---|---|---|
| The written recipe | The learning algorithm | A reproducible procedure, independent of any execution |
| The ingredients | The training dataset | Raw material consumed by the procedure |
| The oven setting and baking time | The hyperparameters | Fixed before execution, by a human, not derived from the material |
| The act of baking | Training | Execution of the procedure on the raw material |
| The resulting cake | The model | A unique, dated result, entirely dependent on the ingredients |
| A slice served to a guest | A prediction | Use of the result on one particular case |
| Sending the recipe to a hungry guest | Deploying the algorithm instead of the model | The guest would have to buy the ingredients and bake it themselves |
| The cake boxed and delivered | The serialized, deployed model | What is actually handed to the consumer |
| Baking a new batch with fresh ingredients | Retraining | Same recipe, new material, new result |
| Two bakers, same recipe, different flour | Two teams, same algorithm, different datasets | Same structure, different parameter values |
| Rewriting the recipe itself | Changing the algorithm or the code | A development act, not a maintenance act |
Limits of the analogy. The cake is a passive object, whereas the model is a function from input to output (section 5.2). A slice once eaten is gone, whereas a model produces an unlimited number of predictions without depleting. And the cake spoils through physical alteration, whereas a model file does not degrade at all — what shifts is the distribution of the production data, moving away from the distribution the model was fitted on (chapter 082).
The code below applies a single algorithm to two datasets describing the same
phenomenon — the price of a home as a function of its floor area, its room count
and its age — on two distinct housing markets. Each dataset holds 800
transactions described by three explanatory variables, with price in dollars.
import pandas as pd
from sklearn.linear_model import LinearRegression
FEATURES = ["area_sqm", "room_count", "age_years"]
north, south = pd.read_csv("market_north.csv"), pd.read_csv("market_south.csv")
X_north, y_north = north[FEATURES], north["price"]
X_south, y_south = south[FEATURES], south["price"]
model_north = LinearRegression().fit(X_north, y_north)
model_south = LinearRegression().fit(X_south, y_south)
print("Same algorithm class:", type(model_north) is type(model_south))
for name, m in [("NORTH", model_north), ("SOUTH", model_south)]:
print(f"{name:6s} intercept = {m.intercept_:>10,.1f} | " +
" | ".join(f"{v} = {c:,.1f}" for v, c in zip(FEATURES, m.coef_)))
listing = pd.DataFrame([{"area_sqm": 85.0, "room_count": 4, "age_years": 12}])
print("North prediction:", round(float(model_north.predict(listing)[0]), 1))
print("South prediction:", round(float(model_south.predict(listing)[0]), 1))Same algorithm class: True
NORTH intercept = 43,934.1 | area_sqm = 3,177.4 | room_count = 8,956.8 | age_years = -887.1
SOUTH intercept = 30,934.4 | area_sqm = 1,935.4 | room_count = 3,768.8 | age_years = -1,481.6
North prediction: 339191.6
South prediction: 192740.5Interpretation. One algorithm class was instantiated, with no hyperparameter changed. The two objects share the same type and the same structure — an affine equation on three variables — and differ only in the values of their four parameters, which come entirely from the data. The gap is a business gap: the northern model values a square meter at roughly $3,177 and the southern one at roughly $1,935, and the discount attached to age is about 1.7 times steeper on the second market. Asked about the same 85 m² four-room, twelve-year-old property, they answer $339,192 and $192,741. Neither is wrong; they encode two different markets.
Direct consequence. "We use a linear regression" does not identify a prediction system. It names the inductive bias that was chosen. The system is identified by the pair (model, training dataset), dated.
The model in the strict sense — the structure and parameters of the final estimator — is not enough to produce a prediction in production. A raw observation must undergo exactly the transformations the training data underwent: imputation with the same replacement values, encoding with the same reference categories, scaling with the same means and standard deviations. Those transformations carry parameters of their own, fitted on the training set. They are part of what was learned, and they must be serialized together with the estimator.
Rigorous definition. Conversion of the complete state of an in-memory object — structure and parameter values — into a persistent byte sequence, allowing it to be reconstituted later in a separate process with functionally identical behavior.
In plain terms. Saving the model to a file so it can be reloaded later,
elsewhere, without retraining. The usual formats are joblib and pickle (Python
ecosystem, version-dependent), ONNX (interoperable across languages and runtimes),
PMML (an XML standard, limited to certain model families), and library-native
formats such as Booster.save_model for XGBoost and LightGBM.
Point of caution. A joblib or pickle file reconstructs arbitrary Python
objects when read. Loading one whose provenance you do not control is equivalent
to executing unverified code. Deserialization under a different library version is
also not guaranteed: the version must be recorded in the metadata and checked at
load time.
import os, joblib, sklearn, numpy as np, pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
df = pd.read_csv("churn.csv")
y = df.pop("churned")
numeric = ["tenure_months", "monthly_bill", "support_calls"]
categorical = ["plan_type"]
pipeline = Pipeline([
("preparation", ColumnTransformer([
("num", Pipeline([("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler())]), numeric),
("cat", OneHotEncoder(handle_unknown="ignore"), categorical)])),
("estimator", RandomForestClassifier(n_estimators=200, max_depth=8,
random_state=42))]).fit(df, y)
artifact = {"pipeline": pipeline, "sklearn_version": sklearn.__version__,
"expected_columns": numeric + categorical, "decision_threshold": 0.42,
"training_date": "2026-08-25"}
joblib.dump(artifact, "churn_model_v3.joblib", compress=3)
print("File size:", round(os.path.getsize("churn_model_v3.joblib")/1024, 1), "KB")
reloaded = joblib.load("churn_model_v3.joblib")
p_memory = pipeline.predict_proba(df.head(5))[:, 1]
p_reloaded = reloaded["pipeline"].predict_proba(df.head(5))[:, 1]
print("Reloaded probabilities:", np.round(p_reloaded, 4))
print("Strictly identical :", np.array_equal(p_memory, p_reloaded))
print("Version and threshold :", reloaded["sklearn_version"],
reloaded["decision_threshold"])File size: 1214.8 KB
Reloaded probabilities: [0.1383 0.3047 0.448 0.418 0.4388]
Strictly identical : True
Version and threshold : 1.8.0 0.42Interpretation. The file weighs about 1.2 MB while the training set holds only 3,200 rows. That volume is the volume of the 200 trees — that is, of the learned parameters — and it depends on the structure chosen, not on the amount of data. Reloading returns strictly identical probabilities: a model is deterministic, so any variation observed in production without retraining points to a difference in the inputs or in the execution environment, never in the model. Finally, the artifact is not reducible to the pipeline. The decision threshold (chapter 062) is a business decision external to the parameters, and both the library version and the ordered list of columns condition whether the load is valid at all.
Rigorous definition. The execution of algorithm A on a dataset D with hyperparameters λ, consisting of selecting from H the hypothesis ĥ that optimizes an explicit criterion evaluated on D, possibly augmented with a regularization term:
ĥ = argmin over h ∈ H of [ (1/n) · Σᵢ L(h(xᵢ), yᵢ) + Ω(h) ]
where L is the loss function and Ω a term penalizing the complexity of the hypothesis. This is the framework of empirical risk minimization (Vapnik). The quantity being minimized is measured on the training sample, not on the population. That gap is the origin of overfitting (chapter 031).
In plain terms. The study phase: the algorithm goes through the corrected examples, adjusts its parameters so as to be wrong as rarely as possible on them, and stops when the criterion no longer improves meaningfully.
What training does not do. It does not modify the hyperparameters, which stay
exactly as supplied. It does not accumulate across successive calls — a second
fit overwrites the previous state entirely, unless partial_fit or warm_start
is used. And it does not evaluate the model it produced, since evaluation requires
data the model has not seen.
Point of caution. The criterion optimized during training (the loss function) and the criterion used to judge business value (the metric) are distinct, and they rarely coincide (chapter 030).
Change any one of these three components and you obtain a different model, with the dataset held constant. This is worth stating plainly because it is often attributed to the data alone: two teams working on identical data, using the same library, will hand over different artifacts if they differ on the penalty term or on the solver.
| Algorithm | Search strategy inside H | Deterministic at constant data |
|---|---|---|
| Linear regression (least squares) | Closed-form solution or matrix decomposition | Yes |
| Logistic regression | Iterative convex numerical optimization | Yes, for a fixed solver and tolerance |
| Decision tree | Greedy search, locally optimal split at each node | Yes, if no random draw of variables |
| Random forest | Trees built on randomly drawn samples and subspaces | No, unless the seed is fixed |
| Gradient boosting | Sequential addition of learners correcting the residuals | No, unless the seed is fixed |
| k-nearest neighbors | No search at all: the training set is memorized | Yes |
| Multilayer perceptron | Stochastic gradient descent by backpropagation | No, unless the seed is fixed and deterministic execution is enforced |
Point of caution. For the algorithms marked "No", two successive training runs
on the same dataset with the same hyperparameters produce two different models.
Fixing a seed (random_state) is what makes the deliverable reproducible, and the
seed belongs in the artifact metadata alongside the library version. Note that
this randomness lives entirely in training. Once selected, the model is
deterministic at prediction time (section 5.2).
Rigorous definition. Evaluation of the function ĥ at a point x of the space of explanatory variables, producing a value ŷ = ĥ(x) in the output space. The observation x need not belong to the training set; the operational value lies precisely in the case where it does not.
In plain terms. Giving an answer for a new case by applying the learned rule. The convention distinguishes y, the actual observed value, from ŷ, the value the model produces; their difference is the residual (chapter 066).
Point of caution. The term carries no temporal connotation. A model that diagnoses a condition from present-day data produces a prediction in the technical sense, even though nothing about the future is at stake.
Three properties characterize this function.
Determinism. Two calls with the same input produce the same output. The model contains no random element at prediction time, even when its training did.
Statelessness. A prediction does not modify the model. Adapting to new data requires retraining, which is a separate operation performed on a separate schedule.
Closure over its input schema. The model accepts only inputs conforming to the schema recorded at training time — same variables, same order, same types, same known categorical levels — and transformations must be applied with the parameters learned on the training set, not recomputed on the incoming data (chapter 028).
Point of caution. The most expensive failure mode is not the runtime error, which is visible and gets fixed within the hour. It is the prediction produced without any error at all from a badly transformed input. A single serialized pipeline, rather than a sequence of steps manually reproduced in the serving code, eliminates that class of failure by construction (chapters 076 to 079).
| Call | Output produced | Shape | Availability |
|---|---|---|---|
predict(X) | Predicted class or numeric value | Vector of length n | All supervised estimators |
predict_proba(X) | Estimated probability per class | n × K matrix, rows summing to 1 | Classifiers with probabilistic output |
decision_function(X) | Unbounded, uncalibrated score | Vector of length n | Margin-based models, SVMs and linear models |
transform(X) | Transformed representation of the inputs | n × p′ matrix | Transformers, not final estimators |
Point of caution. predict applies a 0.5 threshold to the estimated
probability by default. That threshold is an implementation convention, not an
optimum; revising it is one of the strongest levers available (chapters 062
and 029).
Sense 1 — inference in the machine learning sense (serving). The operation of evaluating a trained model on new data to produce predictions. Synonymous with prediction, the term stands in opposition to training and dominates production engineering vocabulary: inference server, inference latency, batch inference.
Sense 2 — inference in the statistical sense. The process of estimating the characteristics of a population from a sample and quantifying the associated uncertainty: point estimation, confidence interval, hypothesis test, significance. This is the tradition of Fisher, Neyman and Pearson.
Where the ambiguity comes from. The two communities use the same word for operations whose objects are opposed: an individual observation on one side, a population parameter on the other. The distinction is developed by Breiman (2001, Statistical Modeling: The Two Cultures) and by Shmueli (2010, To Explain or to Predict?).
Point of caution. A model with excellent predictive performance provides no inferential guarantee whatsoever. A large coefficient establishes neither the significance of the effect nor its causal direction (chapter 016).
| Criterion | Inference, ML sense | Inference, statistical sense |
|---|---|---|
| Object targeted | An individual observation | A population parameter |
| Question asked | What value for this specific case? | What is the value of θ in the population, and with what uncertainty? |
| Output produced | A prediction ŷ | An estimate, a confidence interval, a test decision |
| Quality criterion | Generalization error on unseen data | Properties of the estimator: bias, consistency, coverage |
| Assumptions relied on | i.i.d. observations, stable distribution | A specified generative model, regularity conditions |
| Usual context | Engineering, production, MLOps | Statistics, epidemiology, econometrics |
Recommended phrasing. Use "produce a prediction" or "serve the model" for the first sense, and name the second explicitly as "statistical inference". The cost of the ambiguity is borne downstream, in meetings where a stakeholder reads a coefficient as an effect size.
| Step | Input consumed | Artifact produced | Chapter |
|---|---|---|---|
| Framing | A business question | Task T and metric P made explicit | 012 |
| Dataset construction | Operational sources | Labeled dataset X, y | 005 to 007 |
| Splitting | The full dataset | Train, validation and test subsets | 026, 027 |
| Fitted preparation | Fitted transformers | 022, 023 | |
| Choice of algorithm | Nature of the problem and constraints | Hypothesis space H and inductive bias retained | 049 |
| Hyperparameter selection | Train and validation | Retained configuration λ | 034, 035 |
| Final training | , | Model ĥ | 029 |
| Evaluation | , | Generalization performance figures | 052 to 075 |
| Serialization | Full pipeline and metadata | A versioned artifact file | 081 |
| Serving | Production observations | Predictions | 081 |
| Monitoring | Production stream | Drift alerts, retraining decision | 082 |
How to read the cycle. Two loops of different nature appear. The short loop, between hyperparameter selection and training, explores several hypothesis spaces during development; it runs on the analyst's timescale, in hours or days. The long loop, from monitoring back to the dataset, is triggered by degrading production performance; it runs on the business timescale, in weeks or months. It re-executes the same algorithm on refreshed data to produce a new model, versioned separately. Retraining therefore never consists of modifying the algorithm. It consists of producing a new artifact.
What is deployed is an artifact — a structure and a set of parameter values — produced by running the random forest algorithm on a determined dataset, on a determined date. The algorithm itself sits in the installed library.
Correct formulation : "We deployed the model churn_v3, obtained by training a
random forest on the January to June data."
The algorithm executes a search procedure and retains nothing once it has finished. The induced knowledge lives in the parameters of the model it produced.
Correct formulation : "The algorithm selected, within the hypothesis space, a model whose coefficients indicate a negative association between tenure and churn."
Retraining means re-executing the same algorithm, with the same hyperparameters, on refreshed data. Not one line of code changes. The confusion leads teams to underestimate the real cost of the maintenance cycle, which is a data cost and an operations cost rather than a development cost.
Correct formulation : "The monthly retraining run produces a new version of the model with no change to the training code."
Preparation steps carry parameters fitted on the training data: imputation medians, encoding categories, means and standard deviations. An estimator serialized on its own, fed by transformations reimplemented by hand in the serving layer, produces wrong predictions with no visible error.
Correct formulation : "The serialized artifact is the complete pipeline, from raw data to prediction, together with its metadata."
Depending on the community, the term denotes producing a prediction for one observation, or estimating a population parameter with a measure of uncertainty. Neither the object nor the validity criteria are the same.
Correct formulation : "The inference server produces predictions. Conclusions about the population would require statistical inference, which this system does not provide."
Two runs of the same algorithm on two datasets produce two artifacts whose parameters differ, as section 3.3 shows numerically. They must be evaluated, versioned and monitored separately, even though the code that produced them is identical.
Correct formulation : "The northern and southern markets are served by two models, both fitted with linear regression, versioned and monitored independently."
THE TWO OBJECTS
LEARNING ALGORITHM — a procedure. A : (D, λ) → ĥ ∈ H
Exists before the data, contains no knowledge,
versioned as a software dependency.
MODEL — an artifact. Structure fixed by A and λ,
parameters fitted on D. Exists only after training,
depends entirely on D, versioned as a dated deliverable.
THE THREE COMPONENTS OF A LEARNING ALGORITHM
1. A hypothesis space H which rules are candidates
2. An optimization criterion how their quality is measured on D
3. A search strategy how H is explored
INDUCTIVE BIAS (Mitchell, 1980)
The preferences not derived from the data that make it possible to
choose one generalization among those consistent with D. Without a
bias, no generalization is possible. No Free Lunch (Wolpert, 1996):
no bias is universally superior; choosing an algorithm is betting
on the shape of the phenomenon.
THE TWO OPERATIONS
TRAINING D, λ → ĥ selection within H
PREDICTION x → ŷ = ĥ(x) evaluation of a function
THE MODEL AS A FUNCTION
INPUT (features) → [ MODEL ] → OUTPUT (prediction)
deterministic · stateless · closed over its input schema
THE CULINARY ANALOGY
recipe → algorithm ingredients → training set
oven → hyperparameters baking → training
cake → model slice → prediction
boxed and delivered cake → deployed serialized artifact
TWO SENSES OF THE WORD INFERENCE
ML sense produce a prediction for one observation
Statistical sense estimate a population parameter with
quantified uncertainty
WHAT IS ACTUALLY DEPLOYED
Not the estimator alone, but the complete pipeline (imputation,
encoding, scaling, estimator) and its metadata: library version,
input schema, decision threshold, random seed, date, scope of
validity.
THE VOCABULARY TEST
"The procedure" can be substituted → the algorithm.
"The trained file" can be substituted → the model.Summary statement
The learning algorithm is a procedure that, given a dataset and hyperparameters, selects from a hypothesis space the rule optimizing an explicit criterion; the model is that rule once selected, an artifact made of a structure and fitted parameters, entirely dependent on the data that produced it; training is the selection operation, prediction is the evaluation of the resulting function on a new observation, and what is versioned, serialized and deployed in production is never the algorithm but the model together with its preparation pipeline and its metadata.
Associated quizzes
008.1-quiz-learning-algorithm.md008.2-quiz-model-artifact.md008.3-quiz-distinction-production.md008.4-quiz-training.md008.5-quiz-prediction-inference.md008.6-quiz-full-cycle.mdNext chapter : 009.0-parameters-and-hyperparameters.md