Classification or Regression: Qualifying the Problem

Block 2 — Identifying the problem

Objective

qualify any supervised learning problem as classification or regression from the structure of the target variable alone, handle the five ambiguous configurations that resist this qualification, state what the choice determines across the whole modeling chain, and arbitrate between two competing formulations of one business need.

Estimated duration

40 minutes

Prerequisites

chapters 001 to 009, in particular 004 (supervised learning), 006 (features and target) and 007 (X, y, labels, classes)

Associated quizzes

010.1-quiz-qualification-criterion.md to 010.7-quiz-reformulation.md

1. The qualification criterion: the structure of the codomain

1.1 What does not qualify a problem

The statements below are routinely used to justify the choice between classification and regression. None of them is valid.

The single criterion is the mathematical structure of the set in which the target variable takes its values. The application domain, the algorithm you have in mind, the type of the explanatory variables and the storage format play no part in this qualification.

1.2 The shared formal frame

Supervised learning, defined in chapter 004, induces from a labeled sample a function

f : X → Y

where X is the space of explanatory variables and Y is the codomain, that is, the set of values the target variable can take.

Classification and regression share this frame in full. They also share the training sample, the notion of generalization, the train/test separation (chapter 026) and the risk of overfitting (chapter 031). They differ on one point only: the structure of Y.

1.3 The fundamental fork

Governing statement of the chapter

The problem type is determined by the nature of the target variable. It is determined neither by the business domain, nor by the algorithm under consideration, nor by the type of the explanatory variables.

2. Classification

2.1 Legal and illegal operations on the target

Direct consequence: the error of a classification model is not measured by a gap but by a count of disagreements, formalized in the confusion matrix (chapter 052). On a single prediction there is no "small" or "large" error: the prediction is either correct or incorrect.

2.2 Form of the output

A classifier in fact produces two distinct outputs, and confusing them is a frequent source of error.

Moving from the first to the second requires a decision rule — most often a threshold in the binary case (chapter 062), or the argmax in the multiclass case. Producing an intermediate real-valued score does not turn the problem into a regression: the codomain of the problem remains the finite set of classes, and the score is nothing more than an intermediate quantity.

2.3 Representative examples

3. Regression

3.1 Legal operations and the meaning of error

Direct consequence: in regression the error is graded. Predicting $248,000 for a property that sold for $250,000 is a good result; predicting $90,000 for the same property is a failure. That distinction cannot even be expressed in the classification frame, where both predictions would be equally "wrong".

3.2 Form of the output

A regressor has no equivalent of predict_proba(): there is no "probability of each possible value" when the codomain is continuous. This asymmetry is developed in chapter 029.

3.3 Representative examples and paired examples

The table below places side by side, for one and the same business domain, a target that belongs to classification and a target that belongs to regression. It establishes that the domain determines nothing.

How to read the table: each row describes a single domain, often a single dataset, with the same explanatory variables. Only the column selected as the target changes, and with it the problem type.

4. The qualification question and its decision tree

4.1 The question to ask

What exactly is to be produced, and what is the nature of the value produced: a category drawn from a fixed list, or a number located on a scale?

Its short form, usable in a scoping meeting:

Point of caution: the question must bear on the value produced by the model, not on the decision the organization takes afterwards. A model estimating a loss amount (regression) can feed a binary accept-or-refuse decision. The downstream decision does not change the nature of the model output.

4.2 The three-operation test

When the business question stays ambiguous, the following test settles it mechanically. It applies to the observed values of the target.

Reading rule

• Only operation 1 is meaningful: classification (nominal target).
• Operations 1 and 2 are meaningful, 3 is not: ordinal case, treated in 5.1.
• All three are meaningful: regression.

Applied to a ZIP code: two ZIP codes can be compared for equality; the ordering 33101 < 94110 expresses nothing about the business; the difference 94110 − 33101 = 61,009 is not a distance. The ZIP code is a nominal variable despite its numeric storage.

4.3 The complete decision tree

4.4 Discrete variables and continuous variables

4.5 Instrumented check on the nature of the target

scikit-learn exposes a function that inspects the type of a target array. It is a useful control, provided you know its limits.

import numpy as np
from sklearn.utils.multiclass import type_of_target

targets = {
    "transaction_status   ": np.array(["normal", "fraud", "normal", "fraud"]),
    "iris_species         ": np.array(["setosa", "versicolor", "virginica", "setosa"]),
    "temperature_celsius  ": np.array([18.4, 21.0, 19.7, 23.2]),
    "sale_price_usd       ": np.array([245000.0, 312500.0, 189900.0, 405000.0]),
    "monthly_visits       ": np.array([0, 3, 1, 12]),
    "satisfaction_1_to_5  ": np.array([1, 3, 2, 5]),
    "churned              ": np.array([0, 1, 1, 0]),
    "zip_code             ": np.array([94110, 33101, 60614, 94110]),
}

for name, y in targets.items():
    print(f"{name} dtype={str(y.dtype):8s} -> type_of_target = {type_of_target(y)}")

Output

transaction_status    dtype=<U6      -> type_of_target = binary
iris_species          dtype=<U10     -> type_of_target = multiclass
temperature_celsius   dtype=float64  -> type_of_target = continuous
sale_price_usd        dtype=float64  -> type_of_target = multiclass
monthly_visits        dtype=int64    -> type_of_target = multiclass
satisfaction_1_to_5   dtype=int64    -> type_of_target = multiclass
churned               dtype=int64    -> type_of_target = binary
zip_code              dtype=int64    -> type_of_target = multiclass

Interpretation

The first three lines agree with the business analysis. The next three contradict it:

• sale_price_usd is a regression target, reported as multiclass because the four values, although stored as floats, are whole numbers and few in number;
• monthly_visits is a regression target under the three-operation test, reported as multiclass because it is stored as integers;
• satisfaction_1_to_5 is an ordinal target, and the function does not separate it from a nominal one.

The last two lines agree with the business analysis for the wrong reason. churned is labeled binary only because exactly two distinct values happen to be present in this array. zip_code is indeed nominal, but the function would attach multiclass to any small set of integers, quantities included — it is reacting to the count of distinct values, not to the absence of an order.

Operational conclusion: type_of_target applies a syntactic heuristic based on storage type and the number of distinct values. It detects format inconsistencies; it does not qualify a problem. Qualification remains an act of analysis grounded in domain knowledge.

5. The five ambiguous cases and how to treat them

Five configurations resist direct qualification. Each calls for a documented decision, not a default choice.

5.1 Ordinal target

Nature: a finite set carrying a business-meaningful order, with no interpretable distance between consecutive levels. Examples: a risk level {low, medium, high}, a satisfaction rating from 1 to 5, a tumor stage I to IV, a credit rating from AAA to D.

The dilemma: treating the target as multiclass classification destroys the order information — confusing "high" with "low" and confusing "high" with "medium" count as the same error. Treating it as regression assumes equal spacing between levels, an assumption that is generally false: the severity gap between stages I and II does not equal the gap between III and IV.

Choice adopted: classification, as the qualification of the problem, since the codomain remains finite. Two adjustments are recommended:

Falling back to regression: acceptable when the number of levels is high — from roughly ten upwards — and equal spacing is defensible, the real-valued prediction then being rounded. That choice must be justified explicitly. Ordinal classification is detailed in chapter 011.

5.2 Count target

Nature: the set of natural numbers — countably infinite, ordered, and carrying an interpretable distance. Examples: annual claims, visits, defects per batch.

Choice adopted: regression. All three operations of the test are meaningful, and the codomain is not finite. That the values are integers is not a criterion for classification.

Frequent error: treating a count that happens to be bounded in the observed data (0, 1, 2, 3 claims) as a four-class classification. That formulation forbids the model from ever predicting 4, destroys the order, and makes the residual uninterpretable.

5.3 Duration with censoring

Nature: a duration until an event occurs, observed incompletely for part of the sample — the contract is still active, the patient is still alive, the machine has not yet failed by the end of the observation window.

Choice adopted: neither classification nor regression in the strict sense when censoring is substantial. The table below fixes the course of action.

5.4 A probability as the target

Nature: the value to produce is a number in the interval [0, 1]. Qualification depends entirely on what is observed in the training sample, not on what is requested as output.

Central point: asking for a probability as output does not make a problem a regression. A fraud detection model returning p = 0.83 is still a classification model: the target observed during training is 0 or 1. The continuous output is an estimate of P(Y = 1 | X = x), not a numeric target learned as such.

Point of caution: a probability score produced by a classifier is not necessarily calibrated. A score of 0.83 does not mean that 83% of the cases receiving that score are positive, unless calibration has been verified explicitly (chapter 061).

5.5 A numeric target deliberately discretized

Nature: the available target is numeric — an amount, an age, a rate — and the organization asks for an output in classes defined by cutoffs: "low, medium or high potential customer", "acceptable or unacceptable risk".

Choice adopted: regress on the numeric target, then apply the cutoff to the prediction. Discretizing the target beforehand destroys information and reduces statistical power; the practice is documented in the methodological literature under the name of inappropriate dichotomization (Royston, Altman and Sauerbrei, 2006).

The following protocol quantifies the difference on a customer value dataset. The target is twelve-month revenue. The business defines "high potential" as revenue of at least $8,000. Two models are trained on the same features and the same split: one classifier on the discretized target, one regressor on the raw revenue whose prediction is thresholded at the same cutoff.

import numpy as np
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score, f1_score

rng = np.random.default_rng(7)
n = 8000

# Customer base, features known at the start of the year
tenure_months  = rng.integers(1, 96, n)
past_spend_usd = rng.gamma(2.0, 900, n)
n_orders       = rng.poisson(1 + past_spend_usd / 1800)
support_calls  = rng.poisson(0.8, n)
web_sessions   = rng.poisson(4 + tenure_months / 12)

signal = (1.6 * past_spend_usd + 120 * n_orders + 38 * tenure_months
          + 160 * web_sessions - 300 * support_calls)
revenue_12m = np.maximum(0, signal + rng.normal(0, 1200, n))   # continuous target, USD

X = np.column_stack([tenure_months, past_spend_usd, n_orders, support_calls, web_sessions])

CUTOFF = 8000                                  # "high potential" as the business defines it
y_num = revenue_12m                            # native target
y_cat = (revenue_12m >= CUTOFF).astype(int)    # target destroyed by discretization

X_tr, X_te, yn_tr, yn_te, yc_tr, yc_te = train_test_split(
    X, y_num, y_cat, test_size=0.3, random_state=0, stratify=y_cat)

clf = RandomForestClassifier(n_estimators=300, min_samples_leaf=5, random_state=0).fit(X_tr, yc_tr)
reg = RandomForestRegressor(n_estimators=300, min_samples_leaf=5, random_state=0).fit(X_tr, yn_tr)

pred_clf = clf.predict(X_te)
pred_reg = (reg.predict(X_te) >= CUTOFF).astype(int)

print(f"Test set: {len(yc_te)} customers, {int(yc_te.sum())} high potential at ${CUTOFF:,}\n")
for name, pred in [("discretize the target, then classify", pred_clf),
                   ("regress, then threshold at $8,000", pred_reg)]:
    print(f"{name:36s} precision={precision_score(yc_te, pred):.3f}  "
          f"recall={recall_score(yc_te, pred):.3f}  F1={f1_score(yc_te, pred):.3f}")

print("\nThe business moves the cutoff. The regressor is not retrained:")
for t in (3000, 5000, 12000):
    target_t = (yn_te >= t).astype(int)
    pred_t = (reg.predict(X_te) >= t).astype(int)
    print(f"   cutoff ${t:6,d}: precision={precision_score(target_t, pred_t):.3f}  "
          f"recall={recall_score(target_t, pred_t):.3f}  F1={f1_score(target_t, pred_t):.3f}  "
          f"selected={int(pred_t.sum()):4d}")

Output

Test set: 2400 customers, 548 high potential at $8,000

discretize the target, then classify precision=0.802  recall=0.723  F1=0.760
regress, then threshold at $8,000    precision=0.800  recall=0.706  F1=0.750

The business moves the cutoff. The regressor is not retrained:
   cutoff $ 3,000: precision=0.936  recall=0.963  F1=0.949  selected=2122
   cutoff $ 5,000: precision=0.886  recall=0.873  F1=0.879  selected=1461
   cutoff $12,000: precision=0.714  recall=0.556  F1=0.625  selected=  49

Interpretation

At the cutoff the classifier was trained on, the two formulations are effectively tied: F1 of 0.760 against 0.750, a gap of one point that no practitioner should treat as a decisive advantage. The classifier optimizes directly for that one separation, which buys it a marginal edge there and nothing anywhere else.

The asymmetry appears on the second block. The regressor answers the $3,000, $5,000 and $12,000 questions from the same fitted model, with no relabeling and no retraining, because it learned the underlying quantity rather than one arbitrary split of it. The classifier can answer exactly one question, the one it was trained on; every new cutoff costs a full relabel-and-retrain cycle. The argument for regressing first is therefore not that it wins on the metric at a fixed cutoff — it does not — but that the discretized formulation throws away the target's resolution in exchange for nothing.

Accepted exception: when the underlying numeric target is not observable and only the class is recorded in the source system — a self-declared income band, a risk level assigned by an expert. The available data is then natively categorical, the problem is a classification, and no reformulation is possible.

5.6 Summary table of the ambiguous cases

6. What the choice determines across the whole chain

Qualification happens at step 3 of the project lifecycle (chapter 012). Every later step depends on it.

6.1 Full comparison table

6.2 Two structural points

The loss function is not the metric. The loss is optimized during training and must be differentiable for gradient-based algorithms; the metric evaluates the model and answers a business question. A classifier optimizes log loss and is judged on recall; a regressor optimizes MSE and is judged on MAE. This distinction is developed in chapter 030.

The decision threshold exists only in binary classification. It is a powerful post-training lever: with the model unchanged, moving the threshold shifts the balance between false positives and false negatives. Regression offers no equivalent lever on the model itself; any threshold applied to its output belongs to the downstream decision rule, not to the model.

7. Reformulation: one business need, two possible frames

7.1 The principle

A business need is not a learning problem. It becomes a learning problem the moment a target variable is chosen. When several targets are available to serve the same need, several formulations coexist, possibly in different frames.

The reference case: predictive maintenance. An operator monitors a fleet of instrumented industrial pumps. The need is single: intervene before failure without multiplying pointless shutdowns. Two targets can be constructed from the same history and the same sensors.

7.2 The two formulations, quantified

The protocol below builds a degradation dataset, derives both targets from the same explanatory variables, trains both models, and tunes for each its own decision threshold on a validation split, against an explicit cost function: $800 for a preventive intervention, $12,000 for an unplanned failure.

import numpy as np
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score, mean_absolute_error

rng = np.random.default_rng(42)
n = 6000

# A fleet of instrumented industrial pumps: the SAME features throughout
hours     = rng.uniform(0, 20000, n)                            # cumulative run hours
vibration = 1.2 + 0.00018 * hours + rng.normal(0, 0.35, n)      # mm/s RMS
bearing_t = 55 + 0.0011 * hours + rng.normal(0, 3.0, n)         # bearing temperature, C
delta_p   = 2.6 - 0.00007 * hours + rng.normal(0, 0.20, n)      # pressure drop, bar
n_repairs = rng.poisson(0.5 + hours / 9000)                     # past interventions

wear = 0.55 * vibration + 0.035 * bearing_t - 0.9 * delta_p + 0.00004 * hours
rul = 300 * np.exp(-0.5 * wear) * np.exp(rng.normal(0, 0.22, n))   # remaining days

X = np.column_stack([hours, vibration, bearing_t, delta_p, n_repairs])
y_reg = rul                        # continuous target -> REGRESSION
y_clf = (rul <= 30).astype(int)    # binary target     -> CLASSIFICATION

X_tr, X_te, yr_tr, yr_te, yc_tr, yc_te = train_test_split(
    X, y_reg, y_clf, test_size=0.3, random_state=0, stratify=y_clf)
X_tr, X_va, yr_tr, yr_va, yc_tr, yc_va = train_test_split(
    X_tr, yr_tr, yc_tr, test_size=0.25, random_state=0, stratify=yc_tr)

C_PREVENTIVE, C_FAILURE = 800, 12000   # dollars: planned intervention / unplanned failure

def cost(y_true, alert):
    return int(alert.sum()) * C_PREVENTIVE + int(((y_true == 1) & (alert == 0)).sum()) * C_FAILURE

clf = RandomForestClassifier(n_estimators=300, min_samples_leaf=5, random_state=0).fit(X_tr, yc_tr)
reg = RandomForestRegressor(n_estimators=300, min_samples_leaf=5, random_state=0).fit(X_tr, yr_tr)

p_va, p_te = clf.predict_proba(X_va)[:, 1], clf.predict_proba(X_te)[:, 1]
r_va, r_te = reg.predict(X_va), reg.predict(X_te)

# Each formulation gets its own decision threshold, tuned on cost, on the validation split
thr_a = min(np.arange(0.02, 0.61, 0.01), key=lambda s: cost(yc_va, (p_va >= s).astype(int)))
thr_b = min(np.arange(10, 121, 1),       key=lambda t: cost(yc_va, (r_va <= t).astype(int)))

print(f"Thresholds tuned on validation: A p >= {thr_a:.2f}   B predicted RUL <= {thr_b} days")
print(f"MAE of B on remaining useful life: {mean_absolute_error(yr_te, r_te):.1f} days")
print(f"Reference cost, no alert at all: ${int(yc_te.sum()) * C_FAILURE:,}\n")

for name, alert in [("A - binary classification ", (p_te >= thr_a).astype(int)),
                    ("B - regression on RUL     ", (r_te <= thr_b).astype(int))]:
    print(f"{name} precision={precision_score(yc_te, alert, zero_division=0):.3f}  "
          f"recall={recall_score(yc_te, alert):.3f}  alerts={int(alert.sum()):3d}  "
          f"missed={int(((yc_te == 1) & (alert == 0)).sum()):2d}  cost=${cost(yc_te, alert):,}")

print("\nB - intervention horizon changed with no retraining:")
for h in (60, 90):
    target_h, alert_h = (yr_te <= h).astype(int), (r_te <= h).astype(int)
    print(f"   horizon {h:2d} d: precision={precision_score(target_h, alert_h):.3f}  "
          f"recall={recall_score(target_h, alert_h):.3f}  alerts={int(alert_h.sum()):3d}")

Output

Thresholds tuned on validation: A p >= 0.05   B predicted RUL <= 40 days
MAE of B on remaining useful life: 18.5 days
Reference cost, no alert at all: $2,076,000

A - binary classification  precision=0.431  recall=0.954  alerts=383  missed= 8  cost=$402,400
B - regression on RUL      precision=0.486  recall=0.913  alerts=325  missed=15  cost=$440,000

B - intervention horizon changed with no retraining:
   horizon 60 d: precision=0.900  recall=0.888  alerts=668
   horizon 90 d: precision=0.932  recall=0.927  alerts=979

Interpretation

Read together, the two blocks say something the cost column alone does not. At the horizon the business fixed today, formulation A is the better buy by $37,600 on this test set. Across horizons the business might fix tomorrow, formulation B is the only one that answers without a new training run, and its predicted RUL can also rank machines for scheduling, which a probability above a threshold cannot do as directly. Neither result dominates. The arbitration is a business decision informed by the numbers, not a technical verdict read off them.

7.3 Criteria for arbitrating between the two formulations

Rule of conduct: when both formulations are feasible, build both, evaluate them on the same business cost function and the same test set, then decide on that result. Comparing heterogeneous metrics — an MAE against an F1 — is not an arbitration.

8. Common reasoning mistakes

9. Summary

THE SINGLE QUALIFICATION CRITERION
    The structure of the codomain of the target variable.
    Not the domain, not the algorithm, not the type of the input
    variables, not the storage format.

CLASSIFICATION
    f : X -> {c1, ..., cK}, a FINITE and UNORDERED set.
    Only equality is defined. Error is counted.

REGRESSION
    f : X -> an interval of R, an ORDERED and METRIC set.
    The gap y - y_pred is meaningful. Error is measured.

THE THREE-OPERATION TEST
    Equality only              -> classification
    Equality + order           -> ordinal case, handled as classification
    Equality + order + gap     -> regression

THE FIVE AMBIGUOUS CASES
    Ordinal              -> classification, order-sensitive metric
    Count                -> regression, Poisson loss
    Censored duration    -> survival analysis, or fixed-horizon classification
    Probability          -> follows the OBSERVED label: binary or proportion
    Binned numeric       -> regression, then threshold the prediction

WHAT THE CHOICE DETERMINES NEXT
    Admissible algorithms, loss function, evaluation metrics, form of the
    output, baseline, splitting strategy, error diagnostics, post-training
    tuning lever, production monitoring indicators.

REFORMULATION
    One business need can be framed either way.
    The two formulations are compared on the SAME cost function
    and the SAME test set, never on their native metrics.

Summary statement

A supervised learning problem is a classification when its target takes values in a finite unordered set, and a regression when its target takes values on a numeric scale where the gap between two values is interpretable; this qualification depends exclusively on the nature of the target variable selected, and it then determines the entire modeling chain, from the admissible algorithms through to the production monitoring indicators.

Associated quizzes

• 010.1-quiz-qualification-criterion.md
• 010.2-quiz-classification.md
• 010.3-quiz-regression.md
• 010.4-quiz-decision-tree.md
• 010.5-quiz-ambiguous-cases.md
• 010.6-quiz-consequences-of-the-choice.md
• 010.7-quiz-reformulation.md

Next chapter : 011.0-types-of-classification.md