Explanatory Variables and the Target

Block 1 — Core vocabulary

Objective

define the explanatory variable and the target variable rigorously, master their competing synonyms and the discipline each one comes from, formalize an observation as a feature vector in a space of dimension p, and apply the professional criteria that decide whether an available column may legitimately be submitted to a model.

Estimated duration

40 minutes

Prerequisites

chapters 001 to 005

Associated quizzes

006.1-quiz-features-target-relation.md to 006.7-quiz-case-studies.md

1. The structure of a supervised problem: FEATURES to MODEL to TARGET

1.1 The mental formula

Every supervised learning problem reduces to a single structure, whatever the application domain, the algorithm chosen or the volume of data.

Memorize the formula literally. It carries three methodological commitments.

1. What sits on the left is known at prediction time. A variable placed on the input side that will not be available at the instant the prediction must be produced invalidates the entire setup. This is the first selection criterion, developed in section 6.
2. What sits on the right is what you are trying to obtain. The target is never handed to the model during prediction. Its value is precisely what the model is asked for.
3. The arrow is not a causal relation. It expresses a statistical association exploited for predictive purposes. Chapter 016 treats the distinction between correlation and causation.

1.2 The two operating regimes

The same structure reads differently depending on the phase. This is the most common source of confusion among beginners.

Point of caution. During training the target sits in the dataset just like any other column. Its functional nature is nevertheless radically different. Treating it as an ordinary column, or forgetting to remove it from the inputs, produces a model that predicts perfectly in training and is unusable in production.

2. The explanatory variable

2.1 Definition

2.2 What an explanatory variable is not

2.3 The taxonomy of synonyms

The field aggregates vocabulary from distinct disciplines, each of which coined its own term for the same object. The names below are strictly equivalent in practice; they differ in origin, register and connotation.

Professional recommendation. In writing, use explanatory variable or feature; both are unobjectionable, and feature is the established usage in code and in conversation. When speaking to business stakeholders, input variable or information supplied to the model are understood immediately. Avoid independent variable for the reasons set out in section 4.

2.4 The two senses of the word feature

The term is used in the literature with two different scopes, and confusing them causes real errors.

The gap between the two senses is exactly the scope of feature engineering (chapter 024) and feature selection (chapter 025).

Notational consequence. The number of explanatory variables p in a model is almost never equal to the number of columns in the original file. Saying "my dataset has 14 features" without stating which sense you mean is a reliable source of misunderstanding between practitioners.

3. The target variable

3.1 Definition

3.2 The taxonomy of synonyms

3.3 One target, or several

Point of caution. The presence of several columns that could serve as the target does not license treating them as one target. Two distinct targets define two distinct problems, each demanding its own model, its own metrics and its own validation.

3.4 The target is a construction, not a natural datum

No operational system spontaneously produces a column named churned. That column is the result of a chain of explicit decisions.

Guiding principle. No algorithmic refinement compensates for a badly defined target. Formalizing the target is the first step in the lifecycle of a project (chapter 012), and it belongs to a discussion with the business, not to a technical decision.

4. Dependent and independent variables

4.1 Origin of the terminology

4.2 Why the terminology misleads in machine learning

The most frequent misreading. Concluding that explanatory variables must be independent of one another because they are called "independent". The requirement of low collinearity does exist for certain parametric models, but it has nothing to do with the origin of the term, which refers to independence with respect to the experimental setup, not among variables.

4.3 The role is functional, not intrinsic

The same quantity changes role depending on the question asked. The column does not change; the project does.

4.4 Terminological recommendation

Expected professional stance. Know the synonyms, be able to translate between registers, and be able to explain why independent variable is an inherited term ill-suited to the observational setting of machine learning.

5. Feature vector and feature space

5.1 From row to vector

5.2 Separating X from y in practice

The dataset used below is simulated so that every output in this chapter can be reproduced. It describes 10,000 telecom customers and carries one binary target, churned.

import numpy as np
import pandas as pd

rng = np.random.default_rng(96)
n = 10_000

age = rng.integers(18, 86, n)
tenure_months = rng.integers(1, 121, n)
plan_type = rng.choice(["Mobile", "Fiber", "Fiber+TV"], n, p=[0.42, 0.38, 0.20])
contract = rng.choice(["Monthly", "12_months", "24_months"], n, p=[0.35, 0.38, 0.27])

base_price = np.select([plan_type == "Mobile", plan_type == "Fiber"], [26.0, 48.0], 74.0)
monthly_bill = np.round(np.clip(base_price + rng.normal(0, 7, n), 15.0, 95.0), 2)
support_calls = rng.poisson(1.6, n)
satisfaction = np.clip(np.round(rng.normal(3.6, 1.1, n)), 1, 5)

logit = (-0.7
         - 0.016 * tenure_months
         + 0.24 * support_calls
         - 0.38 * satisfaction
         + 0.013 * monthly_bill
         + np.where(contract == "Monthly", 0.95, 0.0))
churned = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

pd.DataFrame({
    "client_id": [f"C-{i + 1:05d}" for i in range(n)],
    "age": age,
    "tenure_months": tenure_months,
    "plan_type": plan_type,
    "contract": contract,
    "monthly_bill": monthly_bill,
    "support_calls": support_calls,
    "satisfaction": satisfaction,
    "churned": churned,
}).to_csv("telecom_churn.csv", index=False)

The split itself is three lines. Everything that matters is in the decision of what goes into X and what does not.

import pandas as pd

df = pd.read_csv("telecom_churn.csv")

target = "churned"
identifiers = ["client_id"]

X = df.drop(columns=[target] + identifiers)
y = df[target]

print("X :", X.shape)
print("y :", y.shape)
print()
print(X.dtypes)

X : (10000, 7)
y : (10000,)

age                int64
tenure_months      int64
plan_type            str
contract             str
monthly_bill     float64
support_calls      int64
satisfaction     float64
dtype: object

Interpretation. The dataset holds n = 10,000 observations. The matrix X has seven columns, the target and the identifier having been removed. The shape of y, (10000,), confirms that it is a one-dimensional vector and not a single-column matrix. That distinction is the source of warnings and errors in several libraries.

Point of caution. Write df.drop(columns=[target]) before any other transformation. Any preparation step computed on a dataset that still contains the target exposes you to leakage (chapter 028).

5.3 The feature vector of one observation

print(X.iloc[0])

age                     74
tenure_months           72
plan_type           Mobile
contract         12_months
monthly_bill         27.66
support_calls            3
satisfaction           4.0
Name: 0, dtype: object

This row is not yet a vector of $\mathbb{R}^p$: two components are text. Encoding the categorical variables produces the numeric representation actually submitted to the model.

X_encoded = pd.get_dummies(X, columns=["plan_type", "contract"])
X_numeric = X_encoded.astype(float)

print("Columns before encoding:", X.shape[1])
print("Columns after encoding: ", X_encoded.shape[1])
print(list(X_encoded.columns))
print("dtype:", X_numeric.to_numpy().dtype, "| shape:", X_numeric.to_numpy().shape)
print("Feature vector of the first observation:")
print(X_numeric.to_numpy()[0])

Columns before encoding: 7
Columns after encoding:  11
['age', 'tenure_months', 'monthly_bill', 'support_calls', 'satisfaction',
 'plan_type_Fiber', 'plan_type_Fiber+TV', 'plan_type_Mobile',
 'contract_12_months', 'contract_24_months', 'contract_Monthly']
dtype: float64 | shape: (10000, 11)
Feature vector of the first observation:
[74.   72.   27.66  3.    4.    0.    0.    1.    1.    0.    0.  ]

Interpretation. The observation, described at the outset by seven heterogeneous columns, is now a point of ℝ¹¹. The first five components are the original numeric variables; the next six are binary indicators coding the levels of the two categorical variables. The values 1 in positions eight and nine identify a Mobile plan under a 12-month contract.

Two lessons of general scope.

• p is not the number of columns in the file. Here seven columns produce eleven dimensions. With high-cardinality categorical variables the gap becomes considerable: a municipality variable with 400 levels adds 400 dimensions on its own under indicator encoding.
• The scales are heterogeneous. tenure_months ranges from 1 to 120, monthly_bill from 15 to 95, and the indicators are 0 or 1. This heterogeneity has no effect on a decision tree and is decisive for any model based on distances or on gradient descent (chapter 023).

5.4 The feature space

5.5 The model as a function on the feature space

Essential point. The model is defined over the entire space, including the regions where no observation was ever recorded. It will produce a prediction for a feature vector lying outside the domain covered by training, without signaling that the region was never observed. This silent extrapolation is a major source of production failure and is the reason data drift is monitored (chapter 082).

6. Criteria for admitting an explanatory variable

Having a column does not license using it. Four successive filters apply, in this order, before any consideration of performance.

6.1 First criterion: availability at prediction time

Three forms of unavailability, two of which are routinely overlooked.

Point of caution on latency. A model validated on a complete history can collapse in production because one variable arrives forty-eight hours late and therefore shows up as systematically missing at prediction time. The check is not only whether the variable exists, but whether it is effectively available in the calling system, with the required freshness.

The full mechanism by which an unavailable variable damages a project, its subtler forms and the detection protocols are treated in chapter 028.

6.2 Second criterion: absence of leakage

6.3 Third criterion: business plausibility

A variable statistically associated with the target but with no plausible mechanism must be examined before it is retained. Four outcomes are possible.

Point of caution. Business plausibility is a filter, not a proof. A plausible variable may be useless, and a variable whose mechanism is not understood may be genuinely predictive. The criterion serves to identify the variables that need examination, not to settle the matter on its own.

Point of caution on stability. A variable whose distribution or definition shifts over time — a nomenclature overhaul, a change of collection system, a modified business rule — degrades the model without triggering any technical alert. Monitoring these shifts is treated in chapter 082.

6.4 Fourth criterion: legality and ethics

Point of caution. Removing sensitive variables is not enough to produce a fair model. Indirect variables reconstruct the information that was deleted. A fairness audit is conducted on the model's predictions, not on the list of its inputs alone (chapter 080).

6.5 An eight-question control grid

These eight questions come before any statistical measurement. Selection methods based on importance, correlation or performance intervene after this filtering, and are treated in chapter 025.

6.6 Feature quality dominates the choice of algorithm

The observation below is documented by industrial practice and by modeling competitions alike: for a fixed dataset, the gain obtained by enriching the information supplied to the model generally exceeds the gain obtained by substituting one algorithm for another.

The demonstration is reproducible. The data are simulated, so the generating mechanism is known exactly: the price is the product of the floor area and the local price per square meter, discounted for the age of the building. Six noise variables with no link to the target are also present, to reproduce the ordinary condition of a real dataset.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import r2_score

rng = np.random.default_rng(7)
n = 600

area_sqm = rng.uniform(20, 160, n)
local_price_sqm = rng.uniform(2000, 7000, n)
year_built = rng.integers(1900, 2021, n)
building_age = 2024 - year_built

# Generating mechanism: the price is a PRODUCT, discounted for age
price = area_sqm * local_price_sqm * (1 - 0.0025 * building_age) + rng.normal(0, 40_000, n)

noise = pd.DataFrame({f"noise_{j+1}": rng.normal(0, 1, n) for j in range(6)})

S1 = pd.concat([pd.DataFrame({"area_sqm": area_sqm, "year_built": year_built}), noise], axis=1)
S2 = pd.concat([pd.DataFrame({"area_sqm": area_sqm, "year_built": year_built,
                              "local_price_sqm": local_price_sqm}), noise], axis=1)
S3 = S2.copy()
S3["theoretical_value"] = area_sqm * local_price_sqm
S3["building_age"] = building_age
S4 = S3.drop(columns=[f"noise_{j+1}" for j in range(6)])

feature_sets = [
    ("S1 raw, incomplete", S1),
    ("S2 raw, complete", S2),
    ("S3 engineered", S3),
    ("S4 engineered, denoised", S4),
]
models = [
    ("Linear regression", LinearRegression()),
    ("Decision tree", DecisionTreeRegressor(max_depth=6, random_state=0)),
    ("Gradient boosting", HistGradientBoostingRegressor(random_state=0)),
]

print(f"{'Feature set':26s} | {'Model':18s} |  p | R2 test")
print("-" * 66)
for set_name, X in feature_sets:
    X_tr, X_te, y_tr, y_te = train_test_split(X, price, test_size=0.3, random_state=0)
    for model_name, model in models:
        model.fit(X_tr, y_tr)
        r2 = r2_score(y_te, model.predict(X_te))
        print(f"{set_name:26s} | {model_name:18s} | {X.shape[1]:2d} | {r2:7.3f}")

Feature set                | Model              |  p | R2 test
------------------------------------------------------------------
S1 raw, incomplete         | Linear regression  |  8 |   0.543
S1 raw, incomplete         | Decision tree      |  8 |   0.347
S1 raw, incomplete         | Gradient boosting  |  8 |   0.493
S2 raw, complete           | Linear regression  |  9 |   0.898
S2 raw, complete           | Decision tree      |  9 |   0.819
S2 raw, complete           | Gradient boosting  |  9 |   0.927
S3 engineered              | Linear regression  | 11 |   0.946
S3 engineered              | Decision tree      | 11 |   0.917
S3 engineered              | Gradient boosting  | 11 |   0.929
S4 engineered, denoised    | Linear regression  |  5 |   0.947
S4 engineered, denoised    | Decision tree      |  5 |   0.916
S4 engineered, denoised    | Gradient boosting  |  5 |   0.929

What the four feature sets contain

Reading the results

The three lines are, from the topmost at S4 downward: linear regression, then gradient boosting, then the decision tree. The important feature of the chart is not the ranking of the lines but their slope: all three rise steeply from S1 to S3 and then flatten. The horizontal axis, which carries the information supplied, moves performance far more than the vertical spread between models does.

The last two lines are the quantitative heart of the section. On this problem the information lever is roughly four times the algorithm lever.

Why the mechanism produces this result

Interpretation. The generating mechanism is multiplicative. A linear model cannot represent a product of two variables: it is structurally incapable of it for as long as that product is not handed to it. As soon as theoretical_value is constructed, the problem becomes linear again and the simplest model reaches the best performance in the whole table. Gradient boosting, which approximates the product by a sum of piecewise-constant trees, gets part of the way there — at a higher computational cost and with less interpretability.

A second reading, less often stated. Between S1 and S2 the ranking of the models reverses: on S1 the linear model leads (0.543 against 0.493), on S2 gradient boosting leads (0.927 against 0.898). A benchmark conducted on an incomplete feature set can therefore select the wrong algorithm. Comparing models before the features are settled is not merely inefficient; it is unreliable.

Scope of the demonstration and its limits. The result is built on simulated data whose mechanism is known. It illustrates a principle; it is not a generalizable measurement. On real data the gap depends on the problem, and there are documented cases where the change of algorithm dominates — notably when the raw variables already expose the mechanism in a form the hypothesis class can represent, or when the dataset is so large that a flexible model recovers the interaction on its own. The guiding principle nevertheless holds.

Any algorithm whatsoever can exploit only the information contained in the variables it is given. No algorithmic sophistication compensates for absent information.

6.7 The hierarchy of improvement levers

Improving a model is not a menu of independent options. The levers are ordered, and working them out of order wastes budget.

The two levers in the upper-left quadrant — a correctly defined target and the removal of leakage — are cheap and decisive, and they are the two most often skipped. The lever practitioners reach for first, changing the algorithm, sits in the low-effect band.

Point of caution on the ordering. Levers 1 and 2 are not optimizations. They are conditions of validity. A model whose target is badly defined or whose inputs leak does not have a performance number that means anything; tuning it is tuning a measurement instrument that is not connected to the quantity being measured.

7. Three annotated case studies

Each case presents an extract of the raw file, the decisions made on every column, and the specific trap of the domain. The three domains were chosen because the trap differs in each: a consequence column in telecom, a statistic computed too early in real estate, a group structure in industry.

7.1 Telecom churn

Detecting the leak from the completeness rate

The column churn_reason is empty for every customer who did not cancel. A column whose completeness rate coincides exactly with the positive rate of the target is a leaking variable until proven otherwise. The check costs three lines and catches a large share of leakage through a variable.

import numpy as np
import pandas as pd

rng = np.random.default_rng(96)
df = pd.read_csv("telecom_churn.csv")
n = len(df)

# Two columns present in the operational export, populated only for leavers
df["churn_reason"] = np.where(df["churned"] == 1,
                              rng.choice(["Price", "Service", "Competitor"], n), None)
df["churn_date"] = np.where(df["churned"] == 1, "2025-01-14", None)

print("Positive rate of the target :", round(df["churned"].mean(), 4))
print()
print("Completeness rate per column")
print(df.notna().mean().round(4).to_string())

Positive rate of the target : 0.1848

Completeness rate per column
client_id        1.0000
age              1.0000
tenure_months    1.0000
plan_type        1.0000
contract         1.0000
monthly_bill     1.0000
support_calls    1.0000
satisfaction     1.0000
churned          1.0000
churn_reason     0.1848
churn_date       0.1848

Interpretation. Two columns have a completeness rate of 0.1848, equal to the positive rate of the target to four decimal places. That coincidence is not a coincidence: the presence of a value in those columns is the target. No statistical test is needed; the missingness pattern alone convicts them.

The cost of keeping the leak

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

df["has_churn_reason"] = df["churn_reason"].notna().astype(int)

base = ["age", "tenure_months", "monthly_bill", "support_calls", "satisfaction"]
cat = pd.get_dummies(df[["plan_type", "contract"]]).astype(float)

X_clean = pd.concat([df[base], cat], axis=1)
X_leaky = X_clean.copy()
X_leaky["has_churn_reason"] = df["has_churn_reason"]
y = df["churned"]

for name, X in [("Without the leaking column", X_clean),
                ("With has_churn_reason", X_leaky)]:
    X_tr, X_te, y_tr, y_te = train_test_split(
        X, y, test_size=0.3, random_state=0, stratify=y)
    model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
    model.fit(X_tr, y_tr)
    auc = roc_auc_score(y_te, model.predict_proba(X_te)[:, 1])
    print(f"{name:28s} | p = {X.shape[1]:2d} | test AUC = {auc:.4f}")

Without the leaking column   | p = 11 | test AUC = 0.7179
With has_churn_reason        | p = 12 | test AUC = 1.0000

Interpretation. Adding a single leaking indicator takes the AUC from 0.7179, a plausible figure for a churn problem, to 1.0000, a perfect score. Nothing in the run signals a problem: no error, no warning, no unstable behavior. The model is simply reading the answer. In production the column has_churn_reason is empty for every customer who has not yet left — which is every customer the model is asked about — so the model would score them all as non-churners and deliver an AUC near 0.5.

The warning sign to internalize. A perfect or near-perfect score on a problem the business regards as hard is a leakage alert, not a success. The correct reflex is to look for the leaking column before writing the report. Chapter 028 gives the full detection protocol and chapter 092 runs it as a lab.

7.2 Real estate price estimation

Note on energy_rating. The ratings A to G are ordered. Indicator encoding discards that order and spends six dimensions; ordinal encoding preserves it and spends one. The choice is treated in chapter 022; the point here is that the decision belongs to the feature definition stage, not to the modeling stage.

The trap: a statistic computed too early

median_price_sqm_district is simultaneously the strongest performance lever in the problem and a vector of leakage. Computed over the whole dataset, it incorporates the sale prices of the test set into the training inputs. It must be estimated exclusively on the training set, then applied to the other sets, which requires embedding it in a pipeline (chapters 079 and 092) rather than computing it upstream in a notebook cell.

The order of the arrows is the entire content of the diagram. Computing the medians before the split — that is, moving step 4 above step 1 — inflates the measured performance and produces a model that disappoints in production without anyone being able to say why.

Note on temporal dependence. The general level of prices drifts. If the intended use is to estimate future transactions, the split must be chronological (chapter 027) and drift monitoring must be in place (chapter 082). A random split on a market that has risen 15 percent over the period lets the model see the future price level, which is a second, subtler form of leakage.

Note on the ambiguity of rooms. The number of rooms and the floor area are strongly correlated. Keeping both is legitimate for a tree-based model and problematic for a linear model whose coefficients then become unstable and uninterpretable. That is multicollinearity, treated in chapter 017. It affects the reading of the model, not its admissibility as a feature.

7.3 Predictive maintenance on industrial equipment

The trap: the group structure of the observations

A single machine generates thousands of rows that share its signature: its baseline vibration level, its temperature range, its installation era. A random split would place readings from the same machine, and often from the same hour, in both training and test. The model would memorize the machine's signature rather than the degradation mechanism, and the measured performance would be substantially overstated. The split must be conducted by group of machines and must respect chronology (chapter 027).

The trap: window centering

Every rolling window must lie strictly before the observation's timestamp. A window centered on the current instant incorporates future measurements and produces a leak that will only show itself in production.

In pandas, the difference is one argument. rolling(window="24h") on a chronologically sorted, time-indexed frame is trailing and safe; rolling(window="24h", center=True) is centered and leaks. The second call raises no warning.

Note on imbalance. Unplanned failures are rare. A target with a positive rate of one or two percent changes the choice of metric, the reading of the confusion matrix and possibly the sampling strategy. That subject occupies chapters 050 and 051.

7.4 What the three cases have in common

Three different domains, three different traps, one identical discipline: for every column, name the instant at which its value becomes known and compare it with the instant at which the prediction must be produced.

The diagram makes one point that a list of criteria cannot: admission is not permanent. A variable admitted at design time returns to review when its distribution or its definition shifts, which is the link between this chapter and the monitoring covered in chapter 082.

8. Reference sheet

8.1 Formulas and notation

FEATURE VECTOR
    x = (x1, x2, …, xp)ᵀ ∈ ℝ^p
    p = number of components actually submitted to the model, after preparation

DESIGN MATRIX AND TARGET VECTOR
    X ∈ ℝ^(n×p)        row i = xiᵀ, column j = values of variable j
    y ∈ 𝒴^n            𝒴 = ℝ in regression, 𝒴 = {c1, …, ck} in classification

THE MODEL AS A FUNCTION
    f : 𝒳 → 𝒴          defined over the WHOLE space, observed regions included or not
    ŷ = f(x)           the prediction for one observation
    objective          minimize E[ℓ(Y, f(X))] over the distribution, not over the sample

THE ADMISSIBILITY CONDITION
    t_obs ≤ t_pred     for EVERY observation, without exception
    t_obs              instant at which the value of the variable becomes known
    t_pred             instant at which the prediction must be produced

8.2 Reading the dimensions

8.3 Conditions of use, and of non-use

8.4 The scikit-learn call, with its important arguments

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

df = pd.read_csv("telecom_churn.csv")

TARGET = "churned"
IDENTIFIERS = ["client_id"]
LEAKING = []                      # populate after the audit of section 6

X = df.drop(columns=[TARGET] + IDENTIFIERS + LEAKING)
y = df[TARGET]

numeric = X.select_dtypes(include="number").columns.tolist()
categorical = [c for c in X.columns if c not in numeric]

preprocess = ColumnTransformer(
    transformers=[
        ("num", StandardScaler(), numeric),
        ("cat", OneHotEncoder(handle_unknown="ignore", drop=None), categorical),
    ],
    remainder="drop",             # anything not listed is discarded, not passed through
    verbose_feature_names_out=False,
)

pipe = Pipeline([
    ("prep", preprocess),
    ("model", LogisticRegression(max_iter=1000)),
])

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.3, random_state=0, stratify=y)

pipe.fit(X_tr, y_tr)
print("p after preparation :", pipe[:-1].transform(X_tr).shape[1])
print("feature names       :", list(pipe[:-1].get_feature_names_out())[:6], "…")

p after preparation : 11
feature names       : ['age', 'tenure_months', 'monthly_bill', 'support_calls', 'satisfaction', 'plan_type_Fiber'] …

8.5 The eight-question grid, in one block

BEFORE ANY STATISTICAL MEASUREMENT, FOR EVERY CANDIDATE COLUMN
    1. Populated at t_pred, for the whole target population?      no → exclude
    2. Value final at that instant, or revised later?             no → exclude or window
    3. Value a consequence of the event to be predicted?          yes → exclude, leakage
    4. Does an aggregate spill past t_pred?                       yes → recompute, capped window
    5. Plausible business mechanism linking it to the target?     no  → detailed review
    6. Definition and distribution stable over time?              no  → reinforced monitoring
    7. Use lawful given purpose and regulation?                   no  → exclude
    8. Acquisition cost in production acceptable?                 no  → trade-off

9. Common reasoning mistakes

10. Summary

THE MENTAL FORMULA
    FEATURES  →  MODEL  →  TARGET
    what is known        the learned      what is sought
    at instant t_pred    function         and unknown at t_pred

EXPLANATORY VARIABLE — SYNONYMS AND ORIGINS
    feature, characteristic       pattern recognition
    explanatory variable          applied statistics
    independent variable          design of experiments (Fisher, 1935)
    input variable, input         control theory, systems theory
    predictor                     regression statistics
    regressor                     econometrics
    covariate                     biostatistics, clinical trials
    attribute                     data mining, relational databases (Quinlan, 1986)
    descriptor                    chemometrics
    exogenous variable            structural econometrics

TARGET VARIABLE — SYNONYMS AND ORIGINS
    target, target variable       machine learning
    dependent variable            experimental statistics
    response variable             design of experiments
    explained variable            statistics
    endogenous variable           econometrics
    output                        control theory
    label, class                  classification only
    ground truth                  remote sensing, then computer vision
    gold standard                 biostatistics, medical diagnosis
    criterion variable            psychometrics

DEPENDENT / INDEPENDENT TERMINOLOGY
    Inherited from the design of experiments, ill-suited to observational ML.
    "Independent" does not mean independent of the other variables.
    "Dependent" does not mean causally determined.
    The role is functional and reverses from one project to the next.

FORMALIZATION
    x = (x1, …, xp)ᵀ ∈ ℝ^p        feature vector
    X ∈ ℝ^(n×p)                   design matrix
    y ∈ 𝒴^n                       target vector
    f : 𝒳 → 𝒴                     the model, defined over the WHOLE space
    ŷ = f(x)                      the prediction
    p ≠ number of columns in the source file

THE FOUR FILTERS, IN THIS ORDER
    1. Availability at the prediction instant       otherwise exclusion
    2. Absence of leakage                           otherwise exclusion
    3. Business plausibility                        otherwise detailed review
    4. Lawfulness and ethics                        otherwise exclusion
    Statistical selection intervenes only afterwards.

THE FIRST CRITERION
    t_obs ≤ t_pred, for all observations, without exception.
    Three forms of unavailability: temporal, latency, scope.

THE HIERARCHY OF LEVERS
    well-defined target > absence of leakage > new information
    > constructed variables > volume > algorithm > hyperparameters

THE MEASURED EVIDENCE (section 6.6, R² on the test set)
    decision tree on S4, the best features        0.916
    gradient boosting on S1, the worst features   0.493
    algorithm lever, features held fixed at S2    0.108 of spread
    information lever, linear model held fixed    0.404 of spread

Summary statement

The explanatory variables constitute the entirety of the information the model holds and the target variable is the quantity it must estimate; their names vary with the discipline of origin without the notion changing, and the statistical terminology of dependence and independence is an experimental inheritance ill-suited to the observational setting of machine learning. An observation is represented as a vector of $\mathbb{R}^p$ and the model as a function defined over that space, including the regions it never observed. A column becomes an explanatory variable only after satisfying four criteria, the first of which is its effective availability at the instant the prediction will have to be produced; and the quality of that information determines attainable performance more than the choice of algorithm does.

Associated quizzes

• 006.1-quiz-features-target-relation.md
• 006.2-quiz-explanatory-variable.md
• 006.3-quiz-target-variable.md
• 006.4-quiz-dependent-independent.md
• 006.5-quiz-feature-vector-and-space.md
• 006.6-quiz-admission-criteria.md
• 006.7-quiz-case-studies.md

Next chapter : 007.0-x-y-labels-classes.md