The Types of Classification

Block 2 — Identifying the problem
Objective: qualify a classification task rigorously, separate the four canonical types — binary, multiclass, ordinal, multilabel — from the shape and the structure of the target variable alone, know what each type imposes on the choice of algorithm and of metric, and settle once and for all the naming ambiguity carried by logistic regression.
Estimated duration: 55 minutes
Prerequisites: chapters 001 to 010, in particular 007 (X, y, labels, classes) and 010 (classification or regression)
Associated quizzes: 011.1-quiz-task-qualification.md to 011.7-quiz-comparative-summary.md

1. Qualifying a classification task: three discriminating questions

1.1 What chapter 010 settled, and what it left open

Chapter 010 established the first modeling decision: is the target categorical or numeric. Answering "categorical" closes that question and opens another one. It remains to determine which type of classification the task is, because the type governs four things at once:

Governed by the type	Consequence
The shape of the target array	A vector `(n,)` or a matrix `(n, K)`
The set of usable algorithms	Native handling, or decomposition into binary sub-problems
The set of admissible metrics	Accuracy is legal in three of the four types and misleading in all four
The meaning of a single error	A miss, a confusion, a confusion of a given magnitude, or a partially correct answer

A task qualified as "classification" and nothing more is not yet specified enough to be modeled. The rest of this chapter supplies the missing specification.

DEFINITION — Classification task

Rigorous definition

Let X be an input space and C = {c₁, …, $c_K$ } a finite label set of cardinality K ≥ 2. A classification task consists in inducing, from a sample {(xᵢ, yᵢ)}, i = 1…n, a decision function f : X → Y whose codomain Y is built on C by one of two constructions. These two constructions are the foundation of this chapter's partition.

Y = C — every observation receives exactly one label out of K. The problem is single-label.
Y = {0, 1}^K, the power set of C — every observation receives an arbitrary subset of labels, possibly empty. The problem is multilabel.

In plain terms

To classify is to assign labels drawn from a closed list. The first question to settle is whether an observation receives one label or several.

Point of caution

The label set is assumed closed and known at training time. A classifier cannot predict a class it has never observed. Handling previously unseen classes is a different task — anomaly or novelty detection, introduced in chapter 003 and outside the supervised frame of chapter 004.

1.2 The three questions, in order

Order	Question	Answer	Type
1	Can one observation carry several labels at the same time?	Yes	Multilabel
1	Can one observation carry several labels at the same time?	No	Go to question 2
2	How many mutually exclusive classes?	K = 2	Binary
2	How many mutually exclusive classes?	K > 2	Go to question 3
3	Do the classes carry a natural total order?	No	Multiclass (nominal)
3	Do the classes carry a natural total order?	Yes	Ordinal

1.3 Why the order of the questions is not arbitrary

Question 1 comes first because it is the only one that changes the shape of the target array. Answering "yes" turns y from a vector into a matrix, and that change propagates into every downstream API: the fitting call, the prediction call, the metric signatures, the serialization format of the predictions.

Questions 2 and 3 leave the shape untouched. They bear only on the internal structure of C — how many elements it has, and whether it carries an order.

Question	What it changes	What it leaves unchanged
1 — exclusivity	The shape of the target: `(n,)` becomes `(n, K)`	Nothing; every downstream choice is affected
2 — cardinality of C	The number of outputs of the model, the averaging strategy for metrics	The shape of `y`, which stays `(n,)`
3 — order on C	The loss function and the metrics; the cost of a confusion becomes a function of the distance between the true and predicted ranks	The shape of `y`, the number of classes, the algorithms available

An analyst who reverses questions 1 and 2 typically reaches "three classes, therefore multiclass" for a photograph that is simultaneously a beach and a sunset — a specification error that only surfaces when the model is asked to return two labels and cannot.

1.4 The taxonomy at a glance

1.5 The type is a modeling decision, not an observation

One business phenomenon frequently admits several valid formulations. A satisfaction survey scored from 1 to 5 can be handled as an ordinal target with five levels, as a binary target after grouping ({1, 2, 3} against {4, 5}), or as a numeric target under the caveats of chapter 010.

The deciding criterion is the use made of the prediction. If the downstream operational decision is binary — call the customer back or do not — then the binary formulation concentrates the model's capacity on the single boundary that matters.

Formulation of "satisfaction 1 to 5"	Target	When it is the right call
Ordinal, five levels	`(n,)`, ordered	The report distinguishes all five levels, and the cost of an error grows with its magnitude
Binary, detractor against the rest	`(n,)`, 0/1	A single downstream action is triggered, on a single threshold
Multiclass, five nominal levels	`(n,)`, unordered	Almost never: it discards an order the data carries
Regression on the scores	`(n,)`, float	Only if the business is willing to assert that the gaps between consecutive levels are equal

Point of caution: grouping classes is irreversible from the model's standpoint. A model trained on two levels will never restore five-level granularity. Grouping is therefore a decision to be taken with the consumer of the prediction, not a preprocessing convenience.

1.6 What the library can and cannot tell you

scikit-learn exposes type_of_target, which inspects an array and reports what kind of target it looks like. It is a useful format check. It is not a qualification.

python

import numpy as np
from sklearn.utils.multiclass import type_of_target

targets = {
    "binary, 0/1 coded      ": np.array([0, 1, 1, 0, 1]),
    "binary, text labels    ": np.array(["fraud", "normal", "normal", "fraud", "normal"]),
    "multiclass, nominal    ": np.array(["billing", "technical", "sales", "billing", "churn"]),
    "ordinal, rank coded    ": np.array([0, 2, 1, 3, 2]),
    "ordinal, text labels   ": np.array(["low", "high", "medium", "critical", "high"]),
    "multilabel, indicators ": np.array([[1, 1, 0, 0], [0, 0, 1, 0], [1, 0, 1, 1], [0, 0, 0, 0], [1, 1, 1, 0]]),
    "one-hot multiclass     ": np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 0, 1], [0, 1, 0]]),
}

for name, y in targets.items():
    print(f"{name} ndim={y.ndim} -> type_of_target = {type_of_target(y)}")

Output

binary, 0/1 coded       ndim=1 -> type_of_target = binary
binary, text labels     ndim=1 -> type_of_target = binary
multiclass, nominal     ndim=1 -> type_of_target = multiclass
ordinal, rank coded     ndim=1 -> type_of_target = multiclass
ordinal, text labels    ndim=1 -> type_of_target = multiclass
multilabel, indicators  ndim=2 -> type_of_target = multilabel-indicator
one-hot multiclass      ndim=2 -> type_of_target = multilabel-indicator

Interpretation

Two of the four types are invisible to the function.

The two ordinal arrays are reported as multiclass. type_of_target has no access to the order relation, because that relation lives in the domain, not in the array. Ordinal classification has no distinct machine-readable signature.
The one-hot encoded multiclass target is reported as multilabel-indicator, exactly like the genuine multilabel matrix. The function reacts to the shape and to the 0/1 entries, not to the constraint that each row sums to 1.

Operational conclusion: type_of_target separates one-dimensional from two-dimensional targets and counts distinct values. Two of the three discriminating questions of section 1.2 are beyond its reach. The distinction between one-hot multiclass and multilabel can be recovered from the data — the row-sum test of section 5.2 — but the distinction between nominal and ordinal cannot. It has to be asserted by the analyst.

2. Binary classification

DEFINITION — Binary classification

Rigorous definition

The single-label case with K = 2: C = {c₀, c₁} and Y = C. The decision function f : X → {c₀, c₁} partitions the input space into two regions separated by a decision boundary.

Most algorithms do not learn f directly. They learn a score function s : X → ℝ, or an estimate of the conditional probability p(x) = P(Y = c₁ | X = x), and the decision follows from a comparison with a threshold t:

f(x) = c₁ if p(x) ≥ t, and c₀ otherwise.

In plain terms

Two possible answers, and only two. What the model actually produces is a degree of confidence, turned into an answer by fixing a bar.

Point of caution

The separation between estimating p(x) and choosing t is fundamental. The threshold 0.5 is an implementation convention, never a demonstrated optimum. Tuning it is the subject of chapter 062, and section 2.6 below shows what it is worth in dollars.

2.1 The 0/1 convention and what it buys

Convention codes the binary target with the integers 0 and 1. The convention is structural, not decorative: a long chain of downstream behavior depends on it.

Property	Consequence of the 0/1 convention
`y.mean()`	Returns the prevalence of class 1 directly
`y.sum()`	Returns the count of positives
Bernoulli likelihood	Writes as `p^y · (1 − p)^(1 − y)`, the basis of log loss (chapter 061)
`predict_proba`	Returns an `(n, 2)` matrix; column of index 1 carries class 1
`decision_function`	Returns a single column, positive in favor of class 1
`coef_`	Has shape `(1, p)`, a single coefficient vector oriented towards class 1
Confusion matrix	The order `[0, 1]` fixes the layout TN, FP, FN, TP (chapter 052)
Precision, recall, F1	Computed with respect to class 1 by default (`pos_label=1`)
ROC curve, PR curve	Built on the score for class 1 (chapters 063 and 064)

Point of caution: scikit-learn sorts classes in ascending order into the classes_ attribute. With text labels {"Fraud", "Normal"}, alphabetical order places "Fraud" at index 0 — the opposite of the business intent. Coding the target explicitly as 0/1 removes the risk, and section 2.4 measures exactly what it costs to skip that step.

2.2 Worked data — payment card fraud

transaction_id	amount	card_country	merchant_country	hour	channel	is_fraud
T-000001	42.90	FR	FR	14	In-store	0
T-000002	1,890.00	FR	RU	3	Online	1
T-000003	12.50	FR	FR	9	In-store	0
T-000004	7.20	FR	FR	19	Online	0
T-000005	2,450.00	FR	US	4	Online	1

python

import pandas as pd

df = pd.read_csv("transactions.csv")
y = df["is_fraud"]

print("shape           :", y.shape)
print("dtype           :", y.dtype)
print("distinct values :", sorted(y.unique().tolist()))
print("prevalence      :", round(y.mean(), 6))
print()
print(y.value_counts(normalize=True).round(4))

Output

shape           : (284807,)
dtype           : int64
distinct values : [0, 1]
prevalence      : 0.001727

is_fraud
0    0.9983
1    0.0017
Name: proportion, dtype: float64

Interpretation

y is a one-dimensional vector of length n, one value per observation, with two modalities coded 0 and 1. The 0/1 convention pays off immediately: y.mean() returns 0.001727, the prevalence of the positive class, without any further computation.

The positive class accounts for 0.17 % of the observations. The problem is severely imbalanced, which disqualifies accuracy as a steering metric — a constant "never fraud" predictor scores 99.83 % — and calls for the treatments of chapters 050 and 051.

2.3 The positive class and the consequences of choosing it

DEFINITION — Positive class

Rigorous definition

In a binary problem, the positive class is the class with respect to which the counts of the confusion matrix are defined — true positives, false positives, false negatives — and therefore the asymmetric metrics derived from them: precision, recall, specificity, F-beta.

In plain terms

The positive class is the one the model is charged with detecting. The word "positive" carries no favorable connotation: a positive screening result is bad news for the patient.

Designation rule

By professional convention the positive class is the class of interest: the rare, costly or actionable event — fraud, default, disease, churn, equipment failure. In the large majority of cases it coincides with the minority class.

Point of caution

This designation is an explicit modeling decision. Leaving it to the alphabetical order of the labels produces metrics that are correctly computed and read backwards.

Inverting the positive class does not change the model. It changes the meaning of every asymmetric metric computed from it.

Quantity	Effect of inverting the positive class
Learned model, decision boundary	Unchanged
Accuracy	Unchanged — it is a symmetric metric
Precision, recall, F1	Change value: they now describe the other class
Recall and specificity	Swap
False positives and false negatives	Swap
ROC-AUC, score held fixed	Becomes `1 − AUC`
ROC-AUC, score also inverted	Unchanged
PR-AUC	Changes radically: the baseline moves from the prevalence of one class to that of the other

The demonstration below trains one model, then reads it twice.

python

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                             roc_auc_score, average_precision_score,
                             confusion_matrix)

X, y = make_classification(n_samples=4000, n_features=10, n_informative=5,
                           weights=[0.90, 0.10], random_state=0)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=0)

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
y_pred = model.predict(X_test)
score_1 = model.predict_proba(X_test)[:, 1]      # score for class 1

print("confusion matrix, labels=[0, 1]  (rows = truth, columns = prediction)")
print(confusion_matrix(y_test, y_pred, labels=[0, 1]))
print()

for pos in (1, 0):
    print(f"positive class = {pos}   prevalence = {(y_test == pos).mean():.4f}")
    print("  accuracy   :", round(accuracy_score(y_test, y_pred), 4))
    print("  precision  :", round(precision_score(y_test, y_pred, pos_label=pos), 4))
    print("  recall     :", round(recall_score(y_test, y_pred, pos_label=pos), 4))
    print("  pr auc     :", round(average_precision_score(
        (y_test == pos).astype(int), score_1 if pos == 1 else 1 - score_1), 4))
print()
print("ROC AUC, score kept as the class-1 score:")
print("  labels y      :", round(roc_auc_score(y_test, score_1), 4))
print("  labels 1 - y  :", round(roc_auc_score(1 - y_test, score_1), 4))

Output

confusion matrix, labels=[0, 1]  (rows = truth, columns = prediction)
[[1063   12]
 [  48   77]]

positive class = 1   prevalence = 0.1042
  accuracy   : 0.95
  precision  : 0.8652
  recall     : 0.616
  pr auc     : 0.817
positive class = 0   prevalence = 0.8958
  accuracy   : 0.95
  precision  : 0.9568
  recall     : 0.9888
  pr auc     : 0.9889

ROC AUC, score kept as the class-1 score:
  labels y      : 0.9357
  labels 1 - y  : 0.0643

Reading the output, line by line

The confusion matrix holds TN = 1,063, FP = 12, FN = 48, TP = 77. Every figure below is a rearrangement of those four counts.

Metric	Positive class 1	Positive class 0	Relation
Accuracy	0.9500	0.9500	`(TN + TP) / n` is symmetric in the two classes
Recall	0.6160	0.9888	77 / 125 against 1,063 / 1,075 — recall on class 0 is specificity on class 1
Precision	0.8652	0.9568	77 / 89 against 1,063 / 1,111
PR-AUC	0.8170	0.9889	Baselines 0.1042 and 0.8958 — the two numbers are not comparable
ROC-AUC	0.9357	0.0643	Sum exactly 1.0000 when the score is not inverted

The pair 0.9357 / 0.0643 is the sharpest illustration. A model that is genuinely good at ranking frauds first is a model that is genuinely bad at ranking legitimate transactions first, and the two statements are the same statement.

Operational consequence: a recall reported as 0.92 means nothing until the positive class is named. Any report of binary classification results must state it explicitly, next to the number.

ANALOGY — The screening test

A laboratory announces that its test "detects 95 % of cases". The sentence is unusable until you know what is being detected.

If the target is the sick, 95 % is the recall on the class "sick": five sick people in a hundred are missed, and nothing at all has been said about how many healthy people are alarmed for nothing.

If the target is the healthy, 95 % is the recall on the class "healthy" — that is, the specificity — so five healthy people in a hundred are alarmed for nothing, and the rate of missed patients is entirely unknown.

The same number, computed on the same model, describes two unrelated clinical realities. Naming the positive class is the precondition for the number to be interpretable at all.

2.4 The alphabetical-order trap, measured

The trap is not hypothetical and does not raise an error. It silently returns the complementary probability.

python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score

amount = np.array([[12.0], [31.0], [88.0], [240.0], [610.0], [1450.0]])
y_text = np.array(["Normal", "Normal", "Normal", "Fraud", "Fraud", "Fraud"])
y_int = (y_text == "Fraud").astype(int)          # explicit 0 / 1 coding

clf_text = LogisticRegression().fit(amount, y_text)
clf_int = LogisticRegression().fit(amount, y_int)

print("text labels, classes_ :", clf_text.classes_)
print("0/1 labels,  classes_ :", clf_int.classes_)
print()
x_new = np.array([[150.0]])
print("predict_proba column 1, text model :",
      clf_text.predict_proba(x_new)[0, 1].round(4),
      "= P(", clf_text.classes_[1], ")")
print("predict_proba column 1, 0/1 model  :",
      clf_int.predict_proba(x_new)[0, 1].round(4),
      "= P( class", clf_int.classes_[1], ")")
print()
print("recall without pos_label, 0/1 labels  :",
      recall_score(y_int, clf_int.predict(amount)))
try:
    recall_score(y_text, clf_text.predict(amount))
except ValueError as exc:
    print("recall without pos_label, text labels : ValueError")
    print("   ", str(exc).splitlines()[0])

Output

text labels, classes_ : ['Fraud' 'Normal']
0/1 labels,  classes_ : [0 1]

predict_proba column 1, text model : 0.7955 = P( Normal )
predict_proba column 1, 0/1 model  : 0.2045 = P( class 1 )

recall without pos_label, 0/1 labels  : 1.0
recall without pos_label, text labels : ValueError
    pos_label=1 is not a valid label. It should be one of ['Fraud', 'Normal']

Interpretation

Both models are the same model, fitted on the same six rows. The idiom predict_proba(x)[:, 1], written everywhere in tutorials and internal code, returns 0.7955 under text labels and 0.2045 under 0/1 labels. The two numbers sum to 1: they are the probabilities of the two opposite events. A dashboard built on the first would report a 79.6 % fraud risk on a transaction the model scores at 20.5 %.

The final two lines record a second asymmetry. With 0/1 labels the metric functions have a usable default, pos_label=1. With text labels they refuse to guess, and raise. The exception is the good case: it stops the pipeline. The silent column swap above is the bad case.

Rules that remove the class of problem entirely

Rule	Effect
Code the binary target as 0/1, positive event = 1	`classes_` is `[0, 1]`, `pos_label=1` is correct by default
Never index `predict_proba` by a literal 1 without checking `classes_`	Removes the silent swap
Prefer `proba[:, list(model.classes_).index(POSITIVE)]`	Correct whatever the label type
State `pos_label` explicitly in every metric call on text labels	Forces the designation into the code, where it is reviewable

2.5 From score to class: the order of operations

The classification step proper is the comparison with the threshold, downstream of the model. The model itself produces a continuous quantity. That distinction is the whole content of section 6 of this chapter, and the reason a model named "regression" can be a classifier.

The order of operations across the actors involved makes the same point in a different register.

2.6 What the choice is worth in dollars

The positive class is designated because errors on it are expensive. Making that cost explicit changes the ranking of candidate decision rules — and shows that accuracy ranks them backwards.

Cost model for the fraud case: a fraud that is not caught is charged back at an average of $420; a legitimate transaction sent to manual review costs $8 of analyst time.

python

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix

COST_FN = 420   # dollars lost on a fraud that is not caught
COST_FP = 8     # dollars spent reviewing a legitimate transaction

X, y = make_classification(n_samples=4000, n_features=10, n_informative=5,
                           weights=[0.90, 0.10], random_state=0)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=0)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
p1 = model.predict_proba(X_test)[:, 1]

def evaluate(name, y_hat):
    tn, fp, fn, tp = confusion_matrix(y_test, y_hat, labels=[0, 1]).ravel()
    cost = fn * COST_FN + fp * COST_FP
    print(f"{name:<34} acc={accuracy_score(y_test, y_hat):.4f}  "
          f"FP={fp:4d}  FN={fn:4d}  cost=${cost:,}")

evaluate("always predict class 0", np.zeros_like(y_test))
evaluate("model, threshold 0.50", (p1 >= 0.50).astype(int))
evaluate("model, threshold 0.20", (p1 >= 0.20).astype(int))
evaluate("model, threshold 0.05", (p1 >= 0.05).astype(int))

Output

always predict class 0             acc=0.8958  FP=   0  FN= 125  cost=$52,500
model, threshold 0.50              acc=0.9500  FP=  12  FN=  48  cost=$20,256
model, threshold 0.20              acc=0.9275  FP=  60  FN=  27  cost=$11,820
model, threshold 0.05              acc=0.8092  FP= 217  FN=  12  cost=$6,776

Interpretation

From threshold 0.50 down to 0.05, accuracy falls from 0.9500 to 0.8092 while the cost of the errors falls from $20,256 to $6,776 — a reduction of two thirds. Ranked by accuracy, the four rules are ordered 0.50, 0.20, "always 0", 0.05. Ranked by cost, they are ordered 0.05, 0.20, 0.50, "always 0". The two rankings agree on nothing except the trivial predictor.

The full cost curve across the threshold range:

Threshold	False positives	False negatives	Accuracy	Cost
0.02	400	6	0.6617	$5,720
0.05	217	12	0.8092	$6,776
0.10	131	21	0.8733	$9,868
0.20	60	27	0.9275	$11,820
0.30	34	31	0.9458	$13,292
0.50	12	48	0.9500	$20,256
0.70	1	58	0.9508	$24,368
0.90	0	76	0.9367	$31,920

What this establishes for the rest of the chapter: in binary classification the pair (positive class, threshold) is a business decision, not a default. The model supplies the ranking; the decision rule is chosen against a cost function. Chapter 057 develops the precision/recall arbitration and chapter 062 the threshold search.

3. Multiclass classification

DEFINITION — Multiclass classification

Rigorous definition

Single-label classification with K > 2, where the set C = {c₁, …, $c_K$ } is exhaustive — every observation admits a label in C — and mutually exclusive — every observation admits exactly one. Formally Y = C, and $\sum_k$ P(Y = $c_k$ | X = x) = 1 for every x.

The task is also called nominal classification, to emphasize that C carries no order relation.

In plain terms

More than two possible answers, exactly one answer per observation, and no answer is "greater" than another.

Point of caution

Exhaustiveness and exclusivity are two distinct assumptions, and both are falsifiable against the data. A nomenclature that includes an "Other" bucket satisfies exhaustiveness at the price of a heterogeneous class that is hard to model and whose errors are hard to interpret.

3.1 Multiclass is not multilabel

The distinction does not bear on the number of classes. It bears on the number of labels assigned to one observation.

Criterion	Multiclass	Multilabel
Number of possible classes K	K > 2	K ≥ 2
Labels per observation	Exactly 1	0, 1, or several
Mutual exclusivity	Yes, by assumption	No
Shape of the raw `y`	`(n,)`	`(n, K)`
Row sum of the indicator form	Always 1	Between 0 and K
Sum of the predicted probabilities	1, enforced by softmax	Unconstrained
Output layer of a neural network	K units, softmax	K units, independent sigmoids
Usual loss	Categorical cross-entropy	Sum of K binary cross-entropies

The definite-article test: phrase the business question with an article. "What is the reason for this ticket?" is a multiclass problem. "What are the reasons for this ticket?" is a multilabel problem. If the business expert hesitates between the two phrasings, the specification is not settled and no amount of modeling will settle it.

3.2 Worked data — support ticket routing

ticket_id	channel	text_length	premium_customer	tenure_months	category
TK-0001	Email	412	0	14	Billing
TK-0002	Phone	87	1	61	Technical
TK-0003	Web form	235	0	3	Sales
TK-0004	Email	1,104	1	28	Cancellation
TK-0005	Phone	156	0	9	Technical

python

import pandas as pd

tickets = pd.read_csv("tickets.csv")
y = tickets["category"]

print("shape          :", y.shape)
print("distinct labels:", y.nunique())
print()
print(y.value_counts())
print()
print("imbalance ratio:", round(y.value_counts().max() / y.value_counts().min(), 2))

Output

shape          : (12000,)
distinct labels: 4

category
Technical       5184
Billing         3612
Sales           2076
Cancellation    1128
Name: count, dtype: int64

imbalance ratio: 4.6

Interpretation

y is still a vector of shape (n,), exactly as in the binary case. Only the number of modalities changed. Moving from binary to multiclass does not change the structure of y.

The counts range from 1,128 to 5,184, a ratio of 4.6. The imbalance is moderate, and already sufficient to require macro averages rather than micro averages, so that the Cancellation class — the one with the highest business value in a retention context — is not absorbed into the aggregate (chapter 065).

3.3 Decomposition strategies for intrinsically binary algorithms

Some algorithms are binary by construction. Support vector machines (chapter 041) are the canonical example: the maximum-margin formulation separates two classes. Two reduction schemes make it possible to handle K classes with binary classifiers.

DEFINITION — One-vs-Rest and One-vs-One

One-vs-Rest (OvR, also One-vs-All, OvA)

Train K binary classifiers. Classifier k opposes $c_k$ to the union of all the other classes. At prediction time, return argmax_k $s_k$ (x), the class whose classifier gives the highest score.

One-vs-One (OvO)

Train one classifier per pair of classes, that is K(K − 1)/2 classifiers. Classifier (j, k) is trained only on the observations belonging to $c_j$ or $c_k$ . At prediction time, every classifier casts one vote and the majority class wins.

In plain terms

One-vs-Rest asks "is it a cat, yes or no?", then "is it a dog, yes or no?", and keeps the most confident yes. One-vs-One runs a tournament of duels between every pair of classes and counts the wins.

Point of caution

The K OvR scores come from models trained separately, on sub-problems of different difficulty and different prevalence. Their direct comparability under argmax is not guaranteed. That is the acknowledged theoretical weakness of the strategy, and section 3.4 measures its practical footprint.

Criterion	One-vs-Rest (OvR)	One-vs-One (OvO)
Number of classifiers	K	K(K − 1)/2
For K = 4 / 10 / 100	4 / 10 / 100	6 / 45 / 4,950
Size of each sub-problem	n observations	about 2n/K observations
Total cost, algorithm linear in n	O(K · n)	O(K · n)
Total cost, algorithm quadratic in n	O(K · n²)	O(n²)
Induced imbalance	Strong: 1 class against K − 1	None between the two classes of the pair
Ambiguity zones	No positive classifier, or several	Circular votes: A beats B, B beats C, C beats A
Memory at prediction time	K models	K(K − 1)/2 models
scikit-learn default	`LinearSVC`, `SGDClassifier`, `Perceptron`	`SVC`, `NuSVC`

Reading the table: the number of models grows quadratically under OvO, but each model sees only a fraction of the data. For an algorithm whose training cost is quadratic in n — kernel SVM is the standard case — OvO is therefore globally cheaper than OvR despite training more models. Each of the K(K − 1)/2 models pays (2n/K)² instead of n².

The count of models against K makes the growth visible.

The lower line is OvR, growing as K. The upper line is OvO, growing as K(K − 1)/2. At K = 20 the schemes differ by a factor of 9.5 in model count — and OvO can still be the faster of the two, as the measurement below shows.

python

import time
from sklearn.datasets import make_classification
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.svm import SVC

X, y = make_classification(n_samples=6000, n_features=8, n_informative=6,
                           n_classes=6, n_clusters_per_class=1, random_state=0)

t0 = time.perf_counter()
ovr = OneVsRestClassifier(SVC()).fit(X, y)
t_ovr = time.perf_counter() - t0

t0 = time.perf_counter()
ovo = OneVsOneClassifier(SVC()).fit(X, y)
t_ovo = time.perf_counter() - t0

print("K                     :", len(set(y)))
print("OvR sub-models        :", len(ovr.estimators_))
print("OvO sub-models        :", len(ovo.estimators_))
print("OvR fit time, seconds :", round(t_ovr, 2))
print("OvO fit time, seconds :", round(t_ovo, 2))
print("OvR training rows per sub-model :", X.shape[0])
print("OvO training rows per sub-model :", round(2 * X.shape[0] / len(set(y))))

Output

K                     : 6
OvR sub-models        : 6
OvO sub-models        : 15
OvR fit time, seconds : 1.75
OvO fit time, seconds : 0.62
OvR training rows per sub-model : 6000
OvO training rows per sub-model : 2000

Interpretation

OvO trains 15 models against 6, and finishes 2.8 times faster. The arithmetic is in the last two lines: 6 models on 6,000 rows against 15 models on roughly 2,000 rows. With a cost quadratic in n, 6 × 6,000² = 2.16 × 10⁸ against 15 × 2,000² = 6.0 × 10⁷, a predicted ratio of 3.6 — the same order as the 2.8 measured, the gap coming from the fixed overheads that do not scale with n.

This is exactly why scikit-learn chooses OvO for SVC, whose cost is between quadratic and cubic in n, and OvR for LinearSVC, whose cost is linear. The default is not a preference; it is a consequence of the complexity of the base learner.

In both cases the decomposition is internal machinery: y remains a vector (n,), predict returns a single label, and the user code is identical.

3.4 The ambiguity zones, measured

The theoretical weaknesses of both schemes are measurable on ordinary data.

One-vs-Rest: regions where no classifier, or several, say yes.

python

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import LinearSVC

X, y = make_classification(n_samples=3000, n_features=8, n_informative=6,
                           n_classes=5, n_clusters_per_class=1, random_state=3)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=3)

ovr = OneVsRestClassifier(LinearSVC()).fit(X_train, y_train)
scores = ovr.decision_function(X_test)          # shape (n_test, K)
positive_votes = (scores > 0).sum(axis=1)

print("test rows                      :", len(X_test))
print("rows with exactly one positive :", int((positive_votes == 1).sum()))
print("rows with no positive at all   :", int((positive_votes == 0).sum()))
print("rows with two or more positives:", int((positive_votes >= 2).sum()))
print("share in an ambiguity zone     :",
      round(float((positive_votes != 1).mean()), 4))

Output

test rows                      : 900
rows with exactly one positive : 540
rows with no positive and      : 315
rows with two or more positives: 45
share in an ambiguity zone     : 0.4

Interpretation

On 40 % of the test rows, the K binary classifiers do not agree on a single answer: 315 rows where every classifier says "not mine", and 45 rows where at least two claim the observation. In those regions the label returned by predict comes entirely from the argmax of scores produced by models that were never calibrated against one another. The prediction is not wrong by construction — accuracy on this dataset remains usable — but its justification is weaker than the API suggests.

One-vs-One: tied and circular votes.

python

import numpy as np
from itertools import combinations
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.multiclass import OneVsOneClassifier
from sklearn.svm import LinearSVC

X, y = make_classification(n_samples=3000, n_features=8, n_informative=6,
                           n_classes=5, n_clusters_per_class=1, random_state=3)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=3)

ovo = OneVsOneClassifier(LinearSVC()).fit(X_train, y_train)
pairs = list(combinations(sorted(set(y_train)), 2))

votes = np.zeros((len(X_test), len(set(y_train))), dtype=int)
for est, (a, b) in zip(ovo.estimators_, pairs):
    winner = np.where(est.predict(X_test) == 1, b, a)
    votes[np.arange(len(X_test)), winner] += 1

top = votes.max(axis=1)
n_tied = (votes == top[:, None]).sum(axis=1)

print("pairs trained            :", len(pairs))
print("rows with a unique winner:", int((n_tied == 1).sum()))
print("rows with a tied vote    :", int((n_tied > 1).sum()))
print("share of tied votes      :", round(float((n_tied > 1).mean()), 4))
print("example tied vote counts :", votes[np.argmax(n_tied > 1)].tolist())

Output

pairs trained            : 10
rows with a unique winner: 877
rows with a tied vote    : 23
share of tied votes      : 0.0256
example tied vote counts : [2, 3, 1, 1, 3]

Interpretation

2.56 % of test rows produce a tie in the raw vote count. The example row shows [2, 3, 1, 1, 3]: classes 1 and 4 both collect three wins out of four duels, and the majority rule alone cannot separate them. scikit-learn resolves ties by adding a scaled sum of the pairwise decision values to the vote counts, which always breaks them — but the tie-break is a heuristic, invisible in the API and absent from the vote count the user would compute by hand.

Hand-worked: a circular vote on three classes.

Take K = 3 classes A, B, C and one observation x. The three pairwise classifiers return:

Duel	Winner
A against B	A
B against C	B
A against C	C

Vote count: A = 1, B = 1, C = 1. Every class has exactly one win. No majority exists, and no ordering of the three duels produces one either: the relation "beats" is not transitive here. This is Condorcet's paradox, transposed from voting theory to classifier aggregation. With K = 3 the configuration requires a particular geometry of the three boundaries; as K grows, the number of triples that could realize it grows as K(K − 1)(K − 2)/6, and ties become routine.

3.5 Natively multiclass algorithms

Algorithm	Multiclass handling	Chapter
Decision tree	Native — class distribution stored in each leaf	037
Random forest	Native — aggregation of the trees' votes	038
Gradient boosting	Native, usually K ensembles of trees, one per class	039
k-nearest neighbors	Native — majority vote in the neighborhood	040
Naive Bayes	Native — argmax of the posterior probability	042
Logistic regression	Native in the multinomial (softmax) formulation	036
Multilayer perceptron	Native — K output units and a softmax	043
Kernel SVM (`SVC`)	OvO decomposition	041
Linear SVM (`LinearSVC`)	OvR decomposition	041
`RidgeClassifier`	OvR on a ±1 coded target	047

Point of caution: native handling does not exempt you from checking the class distribution. A 12-class problem in which three classes hold 90 % of the mass poses, class by class, exactly the difficulties of an imbalanced binary problem — and the aggregate accuracy hides all of them (chapters 050, 060, 065).

3.6 The shape of the native output, across the three single-label types

python

import numpy as np
from sklearn.datasets import make_classification, make_multilabel_classification
from sklearn.linear_model import LogisticRegression
from sklearn.multioutput import MultiOutputClassifier

# binary
Xb, yb = make_classification(n_samples=500, n_classes=2, random_state=0)
mb = LogisticRegression(max_iter=1000).fit(Xb, yb)

# multiclass
Xm, ym = make_classification(n_samples=500, n_features=10, n_informative=6,
                            n_classes=4, n_clusters_per_class=1, random_state=0)
mm = LogisticRegression(max_iter=1000).fit(Xm, ym)

# multilabel
Xl, Yl = make_multilabel_classification(n_samples=500, n_features=10, n_classes=4,
                                        n_labels=2, random_state=0)
ml = MultiOutputClassifier(LogisticRegression(max_iter=1000)).fit(Xl, Yl)

print("BINARY     y.shape", yb.shape, " coef_", mb.coef_.shape,
      " predict_proba", mb.predict_proba(Xb[:3]).shape)
print("           row sums:", mb.predict_proba(Xb[:3]).sum(axis=1).round(6).tolist())
print()
print("MULTICLASS y.shape", ym.shape, " coef_", mm.coef_.shape,
      " predict_proba", mm.predict_proba(Xm[:3]).shape)
print("           row sums:", mm.predict_proba(Xm[:3]).sum(axis=1).round(6).tolist())
print()
P = np.column_stack([e.predict_proba(Xl[:3])[:, 1] for e in ml.estimators_])
print("MULTILABEL Y.shape", Yl.shape, " sub-models", len(ml.estimators_),
      " per-label P(1)", P.shape)
print("           row sums:", P.sum(axis=1).round(6).tolist())

Output

BINARY     y.shape (500,)  coef_ (1, 20)  predict_proba (3, 2)
           row sums: [1.0, 1.0, 1.0]

MULTICLASS y.shape (500,)  coef_ (4, 10)  predict_proba (3, 4)
           row sums: [1.0, 1.0, 1.0]

MULTILABEL Y.shape (500, 4)  sub-models 4  per-label P(1) [3, 4]
           row sums: [1.790056, 1.456831, 2.24083]

Interpretation

Three structural facts, all readable in the shapes.

Binary stores one coefficient vector, (1, p), not two. The second class is the complement; a second vector would be redundant.
Multiclass stores K coefficient vectors, (K, p), and the softmax normalization forces the rows of predict_proba to sum to 1. Raising the probability of one class necessarily lowers another.
Multilabel stores K separate models. The row sums are 1.79, 1.46, 2.24 — arbitrary. Raising the probability of beach does not lower the probability of sunset, which is the entire point.

The row sum of the predicted probabilities is therefore a runtime signature of the type, exactly as the row sum of the target is a signature of the type in the data.

4. Ordinal classification

DEFINITION — Ordinal scale (Stevens, 1946)

Rigorous definition

In the typology of measurement scales proposed by S. S. Stevens (1946), a scale is ordinal when the order relation between modalities is defined and meaningful, but the distance between two consecutive modalities is not. The admissible operations are comparison (<, >, =), ranks, the median and the quantiles. Sum, arithmetic mean and difference are not admissible.

The interval scale, immediately above, adds meaning to the gaps. The ratio scale adds an absolute zero.

In plain terms

You know how to rank the modalities from weakest to strongest, but you do not know by how much they differ.

Point of caution

Coding an ordinal variable as the integers 0, 1, 2, 3 is a legitimate encoding operation. Treating those integers as measured quantities afterwards is not: the encoding creates a metric that the measurement does not carry.

Scale	Defined relations	Admissible statistics	Example
Nominal	= , ≠	Mode, counts, chi-square	Ticket category, blood group
Ordinal	= , ≠ , < , >	Median, quantiles, rank correlations	Risk grade, tumor stage, Likert item
Interval	= , ≠ , < , > , difference	Mean, standard deviation	Temperature in Celsius, calendar date
Ratio	all of the above, plus ratios	Geometric mean, coefficient of variation	Amount in dollars, duration, count

Ordinal classification is the modeling frame that matches row two of this table: more structure than nominal, less than interval.

DEFINITION — Ordinal classification (ordinal regression)

Rigorous definition

Single-label classification with K > 2, where C carries a total order c₁ ≺ c₂ ≺ … ≺ $c_K$ that is meaningful in the domain, and where no distance is defined on C. The loss function must reflect that order: the cost of confusing c₁ with $c_K$ must exceed the cost of confusing c₁ with c₂.

In plain terms

Classes ranked from weakest to strongest, where being badly wrong is worse than being slightly wrong.

Point of caution

The usual English name is ordinal regression, which sustains a confusion with regression in the sense of chapter 010. The task remains a classification: the codomain is finite. The name is discussed alongside the logistic regression case in section 6.4.

4.1 Worked data — internal credit risk grade

application_id	annual_income_usd	debt_to_income	incidents_12m	tenure_years	risk_level
A-0001	54,000	0.21	0	12	Medium
A-0002	28,500	0.47	1	3	Low
A-0003	19,200	0.63	4	1	Critical
A-0004	41,000	0.38	2	7	High
A-0005	67,300	0.15	0	21	Medium

The order Low ≺ Medium ≺ High ≺ Critical is asserted by the business. No data anywhere states that a Critical file is "twice as risky" as a Medium one.

python

import pandas as pd
from pandas.api.types import CategoricalDtype

applications = pd.DataFrame({
    "application_id": ["A-0001", "A-0002", "A-0003", "A-0004", "A-0005"],
    "annual_income_usd": [54000, 28500, 19200, 41000, 67300],
    "debt_to_income": [0.21, 0.47, 0.63, 0.38, 0.15],
    "incidents_12m": [0, 1, 4, 2, 0],
    "risk_level": ["Medium", "Low", "Critical", "High", "Medium"],
})

risk_scale = CategoricalDtype(
    categories=["Low", "Medium", "High", "Critical"], ordered=True
)
y = applications["risk_level"].astype(risk_scale)

print("dtype         :", y.dtype)
print("ordered       :", y.cat.ordered)
print("rank codes    :", y.cat.codes.tolist())
print("y > 'Medium'  :", (y > 'Medium').tolist())
print("max           :", y.max())
print("sorted        :", y.sort_values().tolist())
try:
    y.mean()
except TypeError as exc:
    print("y.mean()      : TypeError -", str(exc).splitlines()[-1])

Output

dtype         : category
ordered       : True
rank codes    : [1, 0, 3, 2, 1]
y > 'Medium'  : [False, False, True, True, False]
max           : Critical
sorted        : ['Low', 'Medium', 'Medium', 'High', 'Critical']
y.mean()      : TypeError - 'Categorical' with dtype category does not support operation 'mean'

Interpretation

y is still a vector (n,), as in the binary and multiclass cases. The difference is carried by the dtype: ordered=True makes y > 'Medium' valid and interpretable, makes sort_values() return business order rather than alphabetical order, and makes max() return Critical rather than Medium.

The last line is the important one. pandas refuses mean() on an ordered categorical. The refusal is not a limitation; it is the library enforcing Stevens' table. The codes 0 to 3 are ranks, not quantities, and averaging ranks has no defined meaning.

The contrast with an unordered categorical of the same modalities:

python

import pandas as pd
from pandas.api.types import CategoricalDtype

nominal = CategoricalDtype(categories=["Low", "Medium", "High", "Critical"])
z = pd.Series(["Medium", "Low", "Critical"]).astype(nominal)

print("ordered :", z.cat.ordered)
try:
    z > "Medium"
except TypeError as exc:
    print("z > 'Medium' : TypeError -", str(exc).splitlines()[-1])

Output

ordered : False
z > 'Medium' : TypeError - Unordered Categoricals can only compare equality or not

Interpretation

The same four modalities, the same five rows, and the comparison is refused. ordered=True is the only thing that separates a nominal target from an ordinal one in the data structures. It is a declaration made by the analyst — the machine-readable trace of question 3 of section 1.2.

4.2 The two naive treatments and what each costs

Treatment	What it assumes	What is lost or wrongly introduced
As nominal multiclass	No order between the classes	Information loss: every confusion counts the same; predicting `Low` instead of `Critical` costs exactly as much as predicting `High` instead of `Critical`
As regression on codes 0-3	Equal gaps between consecutive levels, and an interval-scale target	Unfounded assumption: nothing establishes that the gap `Low → Medium` equals the gap `High → Critical`; the continuous output also forces arbitrary rounding thresholds
As ordinal	Total order, distances undefined	Treatment consistent with the nature of the measurement

Both naive treatments are used in practice and both are defensible under stated conditions. What is not defensible is using them without stating the condition.

The information discarded by the nominal treatment is directly measurable.

python

import numpy as np
from sklearn.metrics import (accuracy_score, mean_absolute_error,
                             cohen_kappa_score)

# rank codes: 0 = Low, 1 = Medium, 2 = High, 3 = Critical
y_true = np.array([0, 1, 2, 3, 2, 1, 0, 3, 2, 1])
pred_near = np.array([0, 1, 2, 3, 2, 1, 0, 3, 1, 1])   # one error, off by 1 rank
pred_far = np.array([0, 1, 2, 3, 2, 1, 0, 0, 2, 1])    # one error, off by 3 ranks

for name, p in [("near miss  (off by 1 rank)", pred_near),
                ("far miss   (off by 3 ranks)", pred_far)]:
    print(name)
    print("  accuracy        :", accuracy_score(y_true, p))
    print("  MAE on ranks    :", round(mean_absolute_error(y_true, p), 3))
    print("  unweighted kappa:", round(cohen_kappa_score(y_true, p), 3))
    print("  linear kappa    :", round(cohen_kappa_score(y_true, p, weights="linear"), 3))
    print("  quadratic kappa :", round(cohen_kappa_score(y_true, p, weights="quadratic"), 3))

Output

near miss  (off by 1 rank)
  accuracy        : 0.9
  MAE on ranks    : 0.1
  unweighted kappa: 0.865
  linear kappa    : 0.912
  quadratic kappa : 0.952
far miss   (off by 3 ranks)
  accuracy        : 0.9
  MAE on ranks    : 0.3
  unweighted kappa: 0.865
  linear kappa    : 0.737
  quadratic kappa : 0.571

Interpretation

Both predictions get exactly nine of ten rows right. The second one confuses a Critical file with a Low file — the single most expensive error in the domain — and accuracy reports the same 0.90 for both.

Metric	Near miss	Far miss	Sensitive to magnitude?
Accuracy	0.900	0.900	No
Unweighted kappa	0.865	0.865	No
MAE on ranks	0.100	0.300	Yes, linearly
Linear weighted kappa	0.912	0.737	Yes, linearly
Quadratic weighted kappa	0.952	0.571	Yes, quadratically

Two results deserve emphasis. Unweighted Cohen's kappa is as blind as accuracy: correcting for chance agreement does nothing about magnitude, and an analyst who reaches for kappa believing it handles the ordinal case gets no protection at all. And the quadratic weighting separates the two predictions most sharply — 0.952 against 0.571 — because it penalizes by the square of the rank gap: a three-rank error weighs nine times a one-rank error.

Point of caution: MAE on ranks is a model comparison indicator, not a business quantity. It does not read as "0.3 of a risk level on average", because a risk level is not measurable. It reads as "model A is three times further off, on the rank scale, than model B".

4.4 Quadratic weighted kappa, derived by hand

DEFINITION — Quadratic weighted kappa (Cohen, 1968)

Rigorous definition

Let O be the K × K confusion matrix of counts, n the number of observations, rᵢ the i-th row total and $c_j$ the j-th column total. Define the expected matrix under independence, $\mathbb{E}_{ij}$ = rᵢ · $c_j$ / n, and the quadratic weight matrix

$\displaystyle w_{ij} = (i - j)^{2} / (K - 1)^{2}$

The quadratic weighted kappa is

$\displaystyle \kappa_w = 1 - ( \sum_{ij} w_{ij} \cdot O_{ij} ) / ( \sum_{ij} w_{ij} \cdot \mathbb{E}_{ij} )$

Bounds and readings

$\kappa_w$ = 1 for a perfect prediction, since $w_{ii}$ = 0 makes the numerator zero. $\kappa_w$ = 0 for a prediction no better than the chance agreement implied by the marginals. $\kappa_w$ < 0 for a prediction systematically worse than chance; the lower bound depends on the marginals and is not −1 in general.

In plain terms

Compare the average squared rank error you made against the average squared rank error you would have made by guessing at random with the same overall frequency of each grade. Score 1 if you made none, 0 if you did as well as guessing.

Origin and other names

Introduced by Jacob Cohen (1960) unweighted, then generalized with weights by Cohen (1968) to measure inter-rater agreement on ordered categories. It is standard in psychometrics and medical rating studies, and became the default metric of several ordinal machine learning competitions, where it is usually written QWK.

Point of caution

$\kappa_w$ depends on the marginals of the confusion matrix. Two models with identical per-row error patterns but different overall predicted frequencies do not get the same $\kappa_w$ . It is not a decomposable per-observation loss and cannot be optimized directly by gradient descent.

The computation, line by line, on ten observations and three levels

Take K = 3 levels coded 0, 1, 2 and the ten predictions below.

Row	1	2	3	4	5	6	7	8	9	10
$y_{\mathrm{true}}$	0	0	0	1	1	1	1	2	2	2
$y_{\mathrm{pred}}$	0	0	1	1	1	1	2	2	2	1

Step 1 — the observed matrix O, rows indexed by truth, columns by prediction.

        pred 0   pred 1   pred 2     row total
true 0     2        1        0            3
true 1     0        3        1            4
true 2     0        1        2            3
col tot    2        5        3           n = 10

Step 2 — the expected matrix E, $\mathbb{E}_{ij}$ = rᵢ · $c_j$ / n.

E[0][0] = 3 * 2 / 10 = 0.6     E[0][1] = 3 * 5 / 10 = 1.5     E[0][2] = 3 * 3 / 10 = 0.9
E[1][0] = 4 * 2 / 10 = 0.8     E[1][1] = 4 * 5 / 10 = 2.0     E[1][2] = 4 * 3 / 10 = 1.2
E[2][0] = 3 * 2 / 10 = 0.6     E[2][1] = 3 * 5 / 10 = 1.5     E[2][2] = 3 * 3 / 10 = 0.9

Step 3 — the weight matrix W, $w_{ij}$ = (i − j)² / (K − 1)² = (i − j)² / 4.

W[0][0] = 0/4 = 0.00    W[0][1] = 1/4 = 0.25    W[0][2] = 4/4 = 1.00
W[1][0] = 1/4 = 0.25    W[1][1] = 0/4 = 0.00    W[1][2] = 1/4 = 0.25
W[2][0] = 4/4 = 1.00    W[2][1] = 1/4 = 0.25    W[2][2] = 0/4 = 0.00

Step 4 — the weighted observed disagreement.

Σ w·O = 0.00*2 + 0.25*1 + 1.00*0
      + 0.25*0 + 0.00*3 + 0.25*1
      + 1.00*0 + 0.25*1 + 0.00*2
      = 0.25 + 0.25 + 0.25
      = 0.75

Step 5 — the weighted expected disagreement.

Σ w·E = 0.00*0.6 + 0.25*1.5 + 1.00*0.9
      + 0.25*0.8 + 0.00*2.0 + 0.25*1.2
      + 1.00*0.6 + 0.25*1.5 + 0.00*0.9
      = 0.000 + 0.375 + 0.900
      + 0.200 + 0.000 + 0.300
      + 0.600 + 0.375 + 0.000
      = 2.750

Step 6 — the ratio.

κ_w = 1 − 0.75 / 2.75 = 1 − 0.272727… = 0.727272… ≈ 0.7273

Verification against the library

python

import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix

y_true = np.array([0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
y_pred = np.array([0, 0, 1, 1, 1, 1, 2, 2, 2, 1])

O = confusion_matrix(y_true, y_pred)
n = O.sum()
row = O.sum(axis=1)
col = O.sum(axis=0)
E = np.outer(row, col) / n
K = O.shape[0]
i, j = np.indices(O.shape)
W = (i - j) ** 2 / (K - 1) ** 2

print("observed matrix O")
print(O)
print("row totals (truth)     :", row.tolist())
print("column totals (predict):", col.tolist())
print()
print("expected matrix E")
print(E)
print()
print("weight matrix W")
print(W)
print()
print("sum(W * O)  :", round(float((W * O).sum()), 4))
print("sum(W * E)  :", round(float((W * E).sum()), 4))
print("kappa by hand:", round(1 - (W * O).sum() / (W * E).sum(), 4))
print("sklearn QWK  :", round(cohen_kappa_score(y_true, y_pred, weights="quadratic"), 4))

Output

observed matrix O
[[2 1 0]
 [0 3 1]
 [0 1 2]]
row totals (truth)     : [3, 4, 3]
column totals (predict): [2, 5, 3]

expected matrix E
[[0.6 1.5 0.9]
 [0.8 2.  1.2]
 [0.6 1.5 0.9]]

weight matrix W
[[0.   0.25 1.  ]
 [0.25 0.   0.25]
 [1.   0.25 0.  ]]

sum(W * O)  : 0.75
sum(W * E)  : 2.75
kappa by hand: 0.7273
sklearn QWK  : 0.7273

Interpretation

The two figures agree to four decimals. The hand computation and the library call are the same arithmetic. Two properties are worth reading off the matrices directly.

The diagonal of W is zero, so correct predictions contribute nothing to the numerator. Only the confusions are counted, weighted by the square of how far off they were.
E depends only on the marginals. A model that predicts the middle grade for everything gets a column total of 10 on grade 1 and 0 elsewhere, which shrinks Σ w·E and pushes $\kappa_w$ down towards 0 — the correct verdict for a model that has learned nothing but the modal grade.

ANALOGY — The Olympic podium

First, second and third place are strictly ordered. Nobody disputes that first comes before second.

Nothing in that ordering entitles you to write that the gap between first and second equals the gap between second and third. Over 100 meters the winner may beat the runner-up by two hundredths of a second while the runner-up beats third place by half a second.

Treating the places as numbers amounts to asserting that those two gaps are identical. Ignoring the order entirely amounts to asserting that finishing third instead of first is no more regrettable than finishing second. Ordinal classification is the formulation that refuses both errors.

4.5 Approaches that respect the order

The cumulative decomposition turns one ordinal problem into K − 1 binary problems of the form "is y above level k?". The cascade below shows the mechanism as a set of thresholds crossed in sequence.

Recomposing a full distribution over the four levels from the three cumulative probabilities is a matter of differences.

Approach	Principle	Note
Proportional odds model (McCullagh, 1980)	Cumulative logistic model: K − 1 intercepts share one coefficient vector	The statistical reference; the proportional-odds assumption has to be tested
Cumulative binary decomposition (Frank and Hall, 2001)	K − 1 binary classifiers "y > $c_k$ ?", probabilities recomposed by differences	Works with any probabilistic binary classifier
Regression then discretization	Regress on the ranks, then cut with optimized thresholds	Simple and often effective; implicitly assumes equal spacing
Cost-sensitive multiclass	Ordinary multiclass with a cost matrix penalizing large gaps	Reintroduces the order through the loss rather than the model
Ordinal binary decomposition, all-thresholds	K − 1 models sharing a representation, trained jointly	Common in neural network implementations

The Frank and Hall cascade, implemented and read

python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, mean_absolute_error,
                             cohen_kappa_score, confusion_matrix)

rng = np.random.default_rng(0)
n, p = 6000, 6
X = rng.normal(size=(n, p))
beta = np.array([1.4, -1.0, 0.8, 0.0, 0.5, -0.3])
latent = X @ beta + rng.normal(scale=1.0, size=n)
cuts = np.quantile(latent, [0.45, 0.75, 0.92])          # unequal spacing on purpose
y = np.digitize(latent, cuts)                            # ranks 0, 1, 2, 3
K = 4

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=0)


def report(name, y_hat):
    print(name)
    print("  accuracy       :", round(accuracy_score(y_test, y_hat), 4))
    print("  MAE on ranks   :", round(mean_absolute_error(y_test, y_hat), 4))
    print("  quadratic kappa:", round(cohen_kappa_score(y_test, y_hat, weights="quadratic"), 4))


# 1. nominal multiclass baseline
nominal = LogisticRegression(max_iter=1000).fit(X_train, y_train)
report("nominal multiclass", nominal.predict(X_test))

# 2. Frank and Hall cumulative decomposition: K - 1 binary problems "y > k ?"
cumulative = [LogisticRegression(max_iter=1000).fit(X_train, (y_train > k).astype(int))
              for k in range(K - 1)]
p_gt = np.column_stack([m.predict_proba(X_test)[:, 1] for m in cumulative])

proba = np.zeros((len(X_test), K))
proba[:, 0] = 1.0 - p_gt[:, 0]
for k in range(1, K - 1):
    proba[:, k] = p_gt[:, k - 1] - p_gt[:, k]
proba[:, K - 1] = p_gt[:, K - 2]
report("Frank and Hall cascade", proba.argmax(axis=1))

print()
print("row sums of the recomposed probabilities:",
      np.unique(proba.sum(axis=1).round(10)).tolist())
print("negative probabilities produced         :", int((proba < 0).sum()))
print()
print("confusion matrix, Frank and Hall cascade (rows = truth)")
print(confusion_matrix(y_test, proba.argmax(axis=1)))

Output

nominal multiclass
  accuracy       : 0.6744
  MAE on ranks   : 0.3378
  quadratic kappa: 0.7914
Frank and Hall cascade
  accuracy       : 0.6717
  MAE on ranks   : 0.3406
  quadratic kappa: 0.7881

row sums of the recomposed probabilities: [1.0]
negative probabilities produced         : 0

confusion matrix, Frank and Hall cascade (rows = truth)
[[680 126   4   0]
 [147 309  82   2]
 [  6 129 151  20]
 [  1   8  66  69]]

Interpretation

Two things are established and one caution is raised.

The recomposition is coherent: the four probabilities sum to exactly 1 on every row, and no negative probability is produced. That is not guaranteed in general. The differences $q_{k-1}$ − $q_k$ can be negative whenever the K − 1 models are not monotone in k, which happens when they are fitted independently on small samples. Production implementations either isotonic-regress the $q_k$ into a monotone sequence, or share coefficients across the K − 1 models as the proportional-odds model does.

The confusion matrix is concentrated around the diagonal, which is the visual signature of a well-behaved ordinal model. Reading row 4 (Critical): 69 hits, 66 confused with the adjacent High, 8 with Medium, and only 1 with Low. The catastrophic confusion is rare. A nominal model with the same accuracy could have distributed those 75 errors uniformly, and the business consequence would have been entirely different.

The caution: on this dataset, with 4,200 training rows and a correctly specified linear latent structure, the cascade and the nominal model are within one thousandth of each other on all three metrics. The ordinal formulation is not free performance. Its value shows up elsewhere.

Where the ordinal constraint actually pays: small samples

python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, cohen_kappa_score

def run(n_train, seed):
    rng = np.random.default_rng(seed)
    n, p, K = n_train + 3000, 6, 4
    X = rng.normal(size=(n, p))
    beta = np.array([1.4, -1.0, 0.8, 0.0, 0.5, -0.3])
    latent = X @ beta + rng.normal(scale=1.0, size=n)
    y = np.digitize(latent, np.quantile(latent, [0.45, 0.75, 0.92]))
    Xtr, Xte, ytr, yte = train_test_split(X, y, train_size=n_train, test_size=3000,
                                          stratify=y, random_state=seed)
    nom = LogisticRegression(max_iter=2000).fit(Xtr, ytr).predict(Xte)
    cum = [LogisticRegression(max_iter=2000).fit(Xtr, (ytr > k).astype(int)) for k in range(K - 1)]
    pg = np.column_stack([m.predict_proba(Xte)[:, 1] for m in cum])
    pr = np.zeros((len(Xte), K))
    pr[:, 0] = 1 - pg[:, 0]
    for k in range(1, K - 1):
        pr[:, k] = pg[:, k - 1] - pg[:, k]
    pr[:, K - 1] = pg[:, K - 2]
    fh = pr.argmax(axis=1)
    return (cohen_kappa_score(yte, nom, weights="quadratic"),
            cohen_kappa_score(yte, fh, weights="quadratic"),
            mean_absolute_error(yte, nom),
            mean_absolute_error(yte, fh))

for nt in (150, 300, 1000, 4000):
    res = np.array([run(nt, s) for s in range(20)]).mean(axis=0)
    print(f"n_train={nt:5d}  QWK nominal={res[0]:.4f}  QWK cascade={res[1]:.4f}"
          f"   MAE nominal={res[2]:.4f}  MAE cascade={res[3]:.4f}")

Output

n_train=  150  QWK nominal=0.7620  QWK cascade=0.7749   MAE nominal=0.3682  MAE cascade=0.3528
n_train=  300  QWK nominal=0.7854  QWK cascade=0.7923   MAE nominal=0.3407  MAE cascade=0.3326
n_train= 1000  QWK nominal=0.8081  QWK cascade=0.8098   MAE nominal=0.3167  MAE cascade=0.3145
n_train= 4000  QWK nominal=0.8076  QWK cascade=0.8086   MAE nominal=0.3180  MAE cascade=0.3166

Interpretation

Each line averages 20 independent replications, so the ordering is not sampling noise. The cascade's advantage is largest at n = 150 (QWK 0.7749 against 0.7620, MAE 0.3528 against 0.3682) and shrinks steadily to nothing by n = 4,000.

The reason is a parameter count. The nominal model fits K = 4 coefficient vectors, one per class. The cascade fits K − 1 = 3, and each is fitted on the full sample rather than on the rows of one class. The order assumption acts as a constraint on the hypothesis space, which is what regularization is (chapter 033). Like every regularizer, it helps most where data is scarce and fades where data is plentiful.

Metrics for ordinal targets: MAE or RMSE on the ranks (chapters 067 and 069), quadratic weighted kappa, and the confusion matrix read for concentration around the diagonal (chapter 052). Accuracy alone is never sufficient.

4.6 When regression on ranks is defensible

The regression treatment is not forbidden. It is a modeling assumption that must be stated and, where possible, justified.

Condition	Why it matters	How to check
The ordinal levels come from binning a genuine underlying quantity	The equal-spacing assumption is then about the bins, not invented	Ask for the original numeric variable
The bins are of roughly equal width on that quantity	Equal spacing is approximately true	Compare bin boundaries
K is large, 7 or more	The discretization error is small relative to the range	Count the levels
The downstream use is a ranking, not a grade	Only the order of the predictions is consumed	Look at the consuming system
An explicit rounding rule is defined and evaluated	The arbitrary step is made visible and tunable	Optimize the cutoffs on validation data

Where these conditions fail — a four-level risk grade defined by a credit committee, with no underlying measured quantity — the regression treatment asserts something the data does not support, and the assertion should be replaced by a cumulative model or a cost-sensitive multiclass model.

5. Multilabel classification

DEFINITION — Multilabel classification

Rigorous definition

The task whose codomain is Y = {0, 1}^K, the power set of C = {c₁, …, $c_K$ }. Each observation x is associated with a binary vector y = (y₁, …, $y_K$ ) where $y_k$ = 1 if label $c_k$ applies. No constraint bears on $\sum_k$ $y_k$ , which ranges from 0 (no label) to K (all labels).

The labels are neither exclusive nor independent. Modeling their correlations is the difficulty specific to the problem.

In plain terms

One observation can receive several labels at once, or none. Each label is a "yes or no" question that looks independent, but the answers are related to one another.

Point of caution

Multilabel is distinct from multiclass-multioutput, where several target variables are predicted and each is itself multiclass. Multilabel is the special case where every output is binary. scikit-learn separates the two: multilabel- indicator and multiclass-multioutput are different values of type_of_target, and a number of metrics accept the first and refuse the second.

5.1 Worked data — photograph tagging

photo_id	mean_luminance	dominant_hue	faces_detected	beach	sunset	person	animal
P-0001	182	Orange	0	1	1	0	0
P-0002	95	Gray	2	0	0	1	0
P-0003	164	Blue	1	1	0	1	1
P-0004	47	Green	0	0	0	0	0
P-0005	201	Orange	3	1	1	1	0

A photograph can be a beach and a sunset and contain a person, all at once. Photograph P-0004 carries none of the four labels: the zero vector is a valid observation, which is impossible in multiclass.

The relationship between observations and labels is many-to-many, and that is the whole structural difference.

In a multiclass problem the first relation would read PHOTOGRAPH ||--|| LABEL, exactly one label per observation, and the association entity would collapse into a single foreign key on the observation. The TAGGING table with its 0 to K cardinality is the multilabel case in one line of schema.

python

from sklearn.preprocessing import MultiLabelBinarizer

tags = [
    ["beach", "sunset"],
    ["person"],
    ["beach", "person", "animal"],
    [],
    ["beach", "sunset", "person"],
]
mlb = MultiLabelBinarizer(classes=["animal", "beach", "person", "sunset"])
Y = mlb.fit_transform(tags)

print("label order :", mlb.classes_)
print()
print(Y)
print()
print("shape                 :", Y.shape)
print("labels per photograph :", Y.sum(axis=1).tolist())
print("photographs per label :", dict(zip(mlb.classes_, Y.sum(axis=0).tolist())))
print("all-zero rows         :", int((Y.sum(axis=1) == 0).sum()))

Output

label order : ['animal' 'beach' 'person' 'sunset']

[[0 1 0 1]
 [0 0 1 0]
 [1 1 1 0]
 [0 0 0 0]
 [0 1 1 1]]

shape                 : (5, 4)
labels per photograph : [2, 1, 3, 0, 3]
photographs per label : {'animal': 1, 'beach': 3, 'person': 3, 'sunset': 2}
all-zero rows         : 1

Interpretation

Y is a matrix (n, K), no longer a vector. The row sums are 2, 1, 3, 0 — unconstrained. Row 4 is the zero vector, a photograph with no applicable label, and the encoder accepts it without complaint.

Note the explicit classes= argument. Without it, MultiLabelBinarizer infers the label set from the data in sorted order — and any label absent from the training sample simply does not exist in the encoder, silently producing a narrower matrix at fit time than at transform time. Fixing the label list makes the column order part of the contract and reproducible across runs.

5.2 The strict distinction from one-hot multiclass

A multiclass target encoded as indicators also produces an (n, K) matrix. The confusion is frequent; the formal distinction is sharp.

python

from sklearn.preprocessing import LabelBinarizer

lb = LabelBinarizer()
Y_onehot = lb.fit_transform(["Technical", "Billing", "Sales", "Technical", "Cancellation"])

print("label order :", lb.classes_)
print(Y_onehot)
print("row sums    :", Y_onehot.sum(axis=1).tolist())
print("distinct row sums:", sorted(set(Y_onehot.sum(axis=1).tolist())))

Output

label order : ['Billing' 'Cancellation' 'Sales' 'Technical']
[[0 0 0 1]
 [1 0 0 0]
 [0 0 1 0]
 [0 0 0 1]
 [0 1 0 0]]
row sums    : [1, 1, 1, 1, 1]
distinct row sums: [1]

Criterion	Multiclass, one-hot encoded	Multilabel
Matrix shape	`(n, K)`	`(n, K)`
`type_of_target` verdict	`multilabel-indicator`	`multilabel-indicator`
Row sum	Exactly 1	Between 0 and K
Zero vector admissible	No	Yes
Model output	Softmax, probabilities sum to 1	K independent sigmoids
Usual loss	Categorical cross-entropy	Sum of K binary cross-entropies
Recovering the label	`argmax` on the row	Threshold each column independently
Effect of raising P(label 1)	Necessarily lowers another	Leaves the others untouched
Number of fitted models	1	K, or 1 model with K outputs

The recognition test: Y.sum(axis=1) constant and equal to 1 signals an encoded multiclass target; any other distribution signals multilabel. The test is necessary but not sufficient — a multilabel dataset in which every observation happens to carry exactly one label is indistinguishable from a multiclass one on this criterion alone, and the specification has to settle it.

5.3 Decomposition: binary relevance and classifier chains

This is binary relevance: K independent binary classifiers, one per label. It is simple, parallelizable and it scales, but by construction it ignores the correlations between labels. The frequent co-occurrence of beach and sunset is never used.

Classifier chains (Read, Pfahringer, Holmes and Frank, 2011) repair this by appending the predictions of the earlier labels to the feature vector of the later ones.

The measurement below builds a dataset where sunset genuinely depends on beach, and compares the two schemes.

python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.multioutput import MultiOutputClassifier, ClassifierChain
from sklearn.metrics import accuracy_score, hamming_loss, f1_score

rng = np.random.default_rng(4)
n, p = 4000, 8
X = rng.normal(size=(n, p))

beach = (X @ rng.normal(size=p) > 0.3).astype(int)
sunset = ((X @ rng.normal(size=p) > 0.6) | ((beach == 1) & (rng.random(n) < 0.55))).astype(int)
person = (X @ rng.normal(size=p) > 0.0).astype(int)
animal = ((X @ rng.normal(size=p) > 1.2) & (person == 0)).astype(int)
Y = np.column_stack([animal, beach, person, sunset])

Xtr, Xte, Ytr, Yte = train_test_split(X, Y, test_size=0.3, random_state=4)

br = MultiOutputClassifier(LogisticRegression(max_iter=1000)).fit(Xtr, Ytr)
cc = ClassifierChain(LogisticRegression(max_iter=1000), order=[1, 3, 2, 0],
                     random_state=4).fit(Xtr, Ytr)

for name, model in [("binary relevance", br), ("classifier chain", cc)]:
    P = np.asarray(model.predict(Xte)).astype(int)
    print(name)
    print("  subset accuracy :", round(accuracy_score(Yte, P), 4))
    print("  hamming loss    :", round(hamming_loss(Yte, P), 4))
    print("  F1 macro        :", round(f1_score(Yte, P, average="macro", zero_division=0), 4))

print()
print("empirical label co-occurrence, P(sunset = 1 | beach = 1) :",
      round(float(Y[Y[:, 1] == 1, 3].mean()), 4))
print("empirical label co-occurrence, P(sunset = 1 | beach = 0) :",
      round(float(Y[Y[:, 1] == 0, 3].mean()), 4))

Output

binary relevance
  subset accuracy : 0.7625
  hamming loss    : 0.064
  F1 macro        : 0.8968
classifier chain
  subset accuracy : 0.8442
  hamming loss    : 0.0415
  F1 macro        : 0.9525

empirical label co-occurrence, P(sunset = 1 | beach = 1) : 0.7562
empirical label co-occurrence, P(sunset = 1 | beach = 0) : 0.3534

Interpretation

The last two lines quantify the dependence: knowing that beach applies raises the probability of sunset from 0.35 to 0.76. Binary relevance cannot use that information, because its sunset model never sees the beach label.

Chaining recovers it. Subset accuracy rises from 0.7625 to 0.8442, Hamming loss falls from 0.0640 to 0.0415 — a 35 % reduction in the individual label error rate — and macro F1 rises from 0.8968 to 0.9525.

Scheme	Sub-models	Uses label correlation	Parallelizable	Order-dependent
Binary relevance	K	No	Yes, fully	No
Classifier chain	K	Yes, in one direction	No, strictly sequential	Yes
Ensemble of chains	K × B chains	Yes, averaged over orders	Across chains	Averaged away
Label powerset	1 model, up to 2^K classes	Yes, completely	Yes	No

Two implementation cautions. The chain's result depends on the label order — here order=[1, 3, 2, 0] places beach before sunset, which is the informative direction; reversing it would recover much less. And at training time ClassifierChain feeds the true earlier labels to the later models, while at prediction time it feeds the predicted ones. That mismatch, known as exposure bias, means an early-label error propagates down the chain and is a real failure mode on long chains. ClassifierChain accepts cv= to fit the chain on cross-validated predictions instead, which reduces it.

5.4 Metrics: subset accuracy and Hamming loss

A multilabel prediction can be partially correct, a situation that exists in neither binary nor multiclass classification. The metrics have to account for it.

Hand-worked, on five photographs and four labels

	animal	beach	person	sunset	animal	beach	person	sunset	Row exact?	Cells wrong
P1 truth / pred	0	1	0	1	0	1	0	1	Yes	0
P2 truth / pred	0	0	1	0	0	0	1	0	Yes	0
P3 truth / pred	1	1	1	0	1	1	1	1	No	1
P4 truth / pred	0	0	0	1	0	1	0	1	No	1
P5 truth / pred	1	1	1	0	0	1	1	0	No	1

subset accuracy = rows entirely correct / rows
                = 2 / 5
                = 0.40

hamming loss    = wrong individual cells / total cells
                = 3 / (5 × 4)
                = 3 / 20
                = 0.15

The two numbers describe the same predictions. Subset accuracy says 40 %. Hamming loss says 85 % of the individual label decisions are right. Neither is wrong; they answer different questions.

python

import numpy as np
from sklearn.metrics import (accuracy_score, hamming_loss, f1_score,
                             classification_report)

labels = ["animal", "beach", "person", "sunset"]
Y_true = np.array([[0, 1, 0, 1],
                   [0, 0, 1, 0],
                   [1, 1, 1, 0],
                   [0, 0, 0, 1],
                   [1, 1, 1, 0]])
Y_pred = np.array([[0, 1, 0, 1],
                   [0, 0, 1, 0],
                   [1, 1, 1, 1],
                   [0, 1, 0, 1],
                   [0, 1, 1, 0]])

print("rows entirely correct :", int((Y_true == Y_pred).all(axis=1).sum()), "of", len(Y_true))
print("individual cells wrong:", int((Y_true != Y_pred).sum()), "of", Y_true.size)
print()
print("subset accuracy  :", accuracy_score(Y_true, Y_pred))
print("hamming loss     :", round(hamming_loss(Y_true, Y_pred), 4))
print("F1 micro         :", round(f1_score(Y_true, Y_pred, average="micro"), 4))
print("F1 macro         :", round(f1_score(Y_true, Y_pred, average="macro"), 4))
print("F1 samples       :", round(f1_score(Y_true, Y_pred, average="samples"), 4))
print()
print(classification_report(Y_true, Y_pred, target_names=labels, zero_division=0))

Output

rows entirely correct : 2 of 5
individual cells wrong: 3 of 20

subset accuracy  : 0.4
hamming loss     : 0.15
F1 micro         : 0.8571
F1 macro         : 0.831
F1 samples       : 0.8648

              precision    recall  f1-score   support

      animal       1.00      0.50      0.67         2
       beach       0.75      1.00      0.86         3
      person       1.00      1.00      1.00         3
      sunset       0.67      1.00      0.80         2

   micro avg       0.82      0.90      0.86        10
   macro avg       0.85      0.88      0.83        10
weighted avg       0.86      0.90      0.85        10
 samples avg       0.85      0.93      0.86        10

Interpretation

The hand computation and the library agree exactly: 0.40 and 0.15.

The per-label report is where the diagnosis lives. person is predicted perfectly. animal has recall 0.50 — one of the two occurrences was missed, and animal is the rarest label in the sample. beach and sunset are over-predicted, with precision 0.75 and 0.67 against perfect recall. The aggregate figures — micro F1 0.857, macro F1 0.831 — average all of that away.

Note also average="samples", which is specific to multilabel: it computes F1 within each row and then averages over rows, giving 0.8648. It answers "how good is the label set returned for a typical photograph", which is often the question the product actually cares about.

Metric	What it measures	Recommended use
Subset accuracy	Share of rows entirely correct	Very severe; appropriate when the output is consumed as a whole
Hamming loss	Share of individual label decisions that are wrong	Global indicator, tolerant of partial errors
F1 micro	Aggregates the counts across all labels	Dominated by the frequent labels
F1 macro	Mean of the per-label F1	Gives equal weight to rare labels
F1 samples	Mean of the per-row F1	Matches a per-item user experience
F1 per label	Label-by-label detail	Diagnosis; not optional for steering
Coverage error, ranking loss	Quality of the label ranking before thresholding	When the output is a ranked list of suggestions

5.5 The pathology: the model that predicts nothing

Almost every label in a realistic multilabel problem is rare. That fact makes Hamming loss dangerous as a sole steering metric.

python

import numpy as np
from sklearn.metrics import hamming_loss, accuracy_score, f1_score

rng = np.random.default_rng(0)
n, K = 5000, 40
label_rates = rng.uniform(0.005, 0.06, size=K)      # every label is rare
Y_true = (rng.random((n, K)) < label_rates).astype(int)

Y_zero = np.zeros_like(Y_true)                       # predicts nothing, ever

print("labels                       :", K)
print("mean labels per observation  :", round(Y_true.sum(axis=1).mean(), 3))
print("share of positive cells      :", round(Y_true.mean(), 4))
print()
print("degenerate model that always predicts the empty set")
print("  hamming loss    :", round(hamming_loss(Y_true, Y_zero), 4))
print("  subset accuracy :", round(accuracy_score(Y_true, Y_zero), 4))
print("  F1 micro        :", round(f1_score(Y_true, Y_zero, average="micro", zero_division=0), 4))
print("  F1 macro        :", round(f1_score(Y_true, Y_zero, average="macro", zero_division=0), 4))

Output

labels                       : 40
mean labels per observation  : 1.358
share of positive cells      : 0.0339

degenerate model that always predicts the empty set
  hamming loss    : 0.0339
  subset accuracy : 0.2466
  F1 micro        : 0.0
  F1 macro        : 0.0

Interpretation

A model that has learned nothing and returns the empty label set for every photograph achieves a Hamming loss of 0.0339 — 96.6 % of its individual label decisions are correct — and a subset accuracy of 0.2466, because a quarter of the observations genuinely carry no label. Presented on a slide, those two figures look like a working system.

F1 micro and F1 macro are both exactly 0. They are the only two metrics in the list that expose the failure, because both require at least some true positives.

The parallel with the binary imbalanced case of section 2.2 is exact. Hamming loss is to multilabel what accuracy is to imbalanced binary classification: a metric dominated by the majority outcome, which a trivial predictor optimizes.

Point of caution: reading a multilabel model label by label is not optional. Aggregate figures on a sparse label set hide degenerate behavior by construction, and the sparser the labels, the better the degenerate model looks.

6. Logistic regression: a classifier with a misleading name

This is the most frequent naming trap in the field. Logistic regression carries the word "regression" and solves a classification problem.

DEFINITION — Logistic regression

Rigorous definition

A generalized linear model (Nelder and Wedderburn, 1972) for a Bernoulli response, whose link function is the logit. The model posits

$\displaystyle \operatorname{logit}(p(x)) = \ln ( p(x) / (1 - p(x)) ) = \beta_0 + \beta_1x_1 + \dots + \beta_p x_p$

where p(x) = P(Y = 1 | X = x). Inverting the link gives

$\displaystyle p(x) = 1 / (1 + \exp (-(\beta_0 + \beta \cdot x)))$

that is, the logistic function applied to a linear combination of the explanatory variables. The coefficients are estimated by maximizing the likelihood.

In plain terms

A straight line is fitted, not on the class itself, but on the logarithm of the odds of the event; that result is then converted into a probability by an S-shaped curve bounded between 0 and 1.

Point of caution

The model's native output is a probability, not a class. Classification appears only at the next step, by comparison with a threshold (chapter 062). The model is therefore genuinely a regression on a continuous quantity — the probability — in the service of a classification task.

6.1 Where the name comes from

Step	Contribution	Effect on the name
Verhulst, 1838-1845	Introduces the logistic function to model population growth under a carrying capacity	Supplies the adjective "logistic", which names the S-curve
Berkson, 1944	Coins the term logit and promotes the model in biostatistics	Establishes the logit link as standard
Cox, 1958	Formalizes the analysis of binary data with this model	Spreads it through applied statistics
Nelder and Wedderburn, 1972	Unifying frame of generalized linear models	Files the model in the "regression" family, by descent from linear regression
McFadden, 1974-1975	Random utility formulation of discrete choice	Same model, different disciplinary name

The word "regression" is therefore an inheritance from the model's statistical family — a linear model fitted on a transformation of the conditional expectation — not a description of the task it solves. The word "logistic" comes from a demographic growth curve of the 1830s and describes the shape of the link function, nothing more.

6.2 The mechanism, step by step

The regression is real, and it happens at the second box. What it regresses is not the class; it is the log-odds. The classification is the last box, and it is not part of the model at all.

python

import numpy as np
from sklearn.linear_model import LogisticRegression

amount = np.array([[10.0], [20.0], [30.0], [40.0], [50.0], [60.0]])
y = np.array([0, 0, 0, 1, 1, 1])

clf = LogisticRegression().fit(amount, y)

print("classes_        :", clf.classes_)
print("intercept b0    :", clf.intercept_.round(4))
print("coefficient b1  :", clf.coef_.round(4))
print()
query = np.array([[25.0], [35.0], [45.0]])
z = clf.decision_function(query)
print("linear score z  :", z.round(4))
print("sigmoid of z    :", (1 / (1 + np.exp(-z))).round(4))
print("predict_proba   :")
print(clf.predict_proba(query).round(4))
print("predict         :", clf.predict(query))

Output

classes_        : [0 1]
intercept b0    : [-19.7837]
coefficient b1  : [[0.5652]]

linear score z  : [-5.6525 -0.      5.6525]
sigmoid of z    : [0.0035 0.5    0.9965]
predict_proba   :
[[0.9965 0.0035]
 [0.5    0.5   ]
 [0.0035 0.9965]]
predict         : [0 0 1]

Interpretation, line by line

The fitted model is z = −19.7837 + 0.5652 · amount. That is a linear regression, performed on the log-odds of class 1. The coefficient reads as: one additional unit of amount multiplies the odds of fraud by exp(0.5652) = 1.76.

decision_function returns z directly — the raw regression output, unbounded. Applying the sigmoid by hand reproduces column 1 of predict_proba exactly: 0.0035, 0.5, 0.9965. There is no hidden step; predict_proba is the sigmoid of decision_function.

The middle query point deserves attention. At amount = 35 the model sits at the midpoint of its S-curve, z ≈ 0 and p ≈ 0.5, and predict returns class 0. The displayed 0.5 0.5 is a rounding artifact: the exact values are 0.50000397 and 0.49999603, so the comparison with the 0.5 threshold falls on the negative side. Never read a decision off a rounded probability. And where an exact tie does occur, scikit-learn resolves it by argmax over the columns of predict_proba, which returns the class of lowest index — that is, classes_[0].

6.3 What scikit-learn says

python

from sklearn.base import is_classifier, is_regressor
from sklearn.linear_model import (LogisticRegression, LinearRegression,
                                  RidgeClassifier, SGDClassifier)
from sklearn.svm import LinearSVC, LinearSVR
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor

estimators = [LogisticRegression(), LinearRegression(), RidgeClassifier(),
              SGDClassifier(), LinearSVC(), LinearSVR(),
              KNeighborsClassifier(), KNeighborsRegressor()]

print(f"{'estimator':<24}{'classifier':<12}{'regressor':<12}{'predict_proba':<15}{'module'}")
for est in estimators:
    print(f"{type(est).__name__:<24}"
          f"{str(is_classifier(est)):<12}"
          f"{str(is_regressor(est)):<12}"
          f"{str(hasattr(est, 'predict_proba')):<15}"
          f"{type(est).__module__.rsplit('.', 1)[0]}")

Output

estimator               classifier  regressor   predict_proba  module
LogisticRegression      True        False       True           sklearn.linear_model
LinearRegression        False       True        False          sklearn.linear_model
RidgeClassifier         True        False       False          sklearn.linear_model
SGDClassifier           True        False       False          sklearn.linear_model
LinearSVC               True        False       False          sklearn.svm
LinearSVR               False       True        False          sklearn.svm
KNeighborsClassifier    True        False       True           sklearn.neighbors
KNeighborsRegressor     False       True        False          sklearn.neighbors

Interpretation

Rows 1 and 2 settle the question. LogisticRegression and LinearRegression sit in the same module, sklearn.linear_model, because they belong to the same mathematical family. They are separated by is_classifier / is_regressor, which report the task. The module placement records the algorithmic kinship; the interface records the task.

Rows 3 and 4 add a warning about the third column. RidgeClassifier and SGDClassifier are classifiers with no predict_proba. The first has none because it is a ridge regression on a ±1 coded target followed by a sign, and there is no probability in that construction. The second has none with its default loss: SGDClassifier(loss="hinge") reports False, while SGDClassifier(loss="log_loss") reports True. The presence of predict_proba is therefore evidence of a classifier, but its absence is not evidence of a regressor.

The reliable test is is_classifier, which inspects the estimator's declared type, not its method surface.

6.4 Other names to watch

Name	Actual nature	Note
`LogisticRegression`	Classifier	Binary, or multiclass in the multinomial formulation
`RidgeClassifier`	Classifier	Ridge regression on a ±1 coded target, then the sign
`SGDClassifier`	Classifier	Names an optimizer, not a model family; the loss determines the model
`LinearSVC`	Classifier	The "C" is for Classifier; `LinearSVR` is the regression variant
`LinearRegression`	Regressor	Continuous numeric target
Ordinal regression	Classifier	The established term for ordinal classification, section 4
Softmax regression	Classifier	Another name for multinomial logistic regression
Poisson regression	Regressor	Count target, chapter 010
Cox regression	Neither, strictly	Survival analysis; models a hazard rate, chapter 010
`KNeighborsClassifier` / `KNeighborsRegressor`	Classifier / Regressor	One algorithm, two tasks, two distinct classes
`GradientBoostingClassifier` / `...Regressor`	Classifier / Regressor	Same, chapter 039

Professional rule: the name of a class records its mathematical family or its history; only the nature of the target variable determines the task. Where there is doubt, is_classifier settles it.

7. The four types side by side

7.1 Recap table

Criterion	Binary	Multiclass	Ordinal	Multilabel
Number of classes	K = 2	K > 2	K > 2	K ≥ 2
Labels per observation	1	1	1	0 to K
Order on the classes	Not applicable	No	Yes, without distance	Not applicable
Shape of `y`	`(n,)`	`(n,)`	`(n,)`, ordered dtype	`(n, K)` binary
Native model output	1 probability	K softmax probabilities	Cumulative probabilities, or a rank	K independent probabilities
Sum of the probabilities	1	1	1 after recomposition	Unconstrained
`coef_` shape, linear model	`(1, p)`	`(K, p)`	`(K−1, p)` cumulative	`(1, p)` per label
`type_of_target` verdict	`binary`	`multiclass`	`multiclass` — indistinguishable	`multilabel-indicator`
Example	Fraudulent transaction or not	Routing a ticket to a team	Credit risk grade	Tagging a photograph
Appropriate metrics	Precision, recall, F1, ROC-AUC, PR-AUC	Accuracy, macro / micro / weighted F1, confusion matrix	MAE on ranks, quadratic weighted kappa, confusion matrix	Hamming loss, F1 micro / macro / samples, per-label F1
Metric to avoid	Accuracy under marked imbalance	Accuracy under marked imbalance	Accuracy alone: blind to magnitude; unweighted kappa likewise	Hamming loss alone, subset accuracy alone
Trivial predictor that scores well	Always the majority class	Always the majority class	Always the modal grade	Always the empty set
Cross-references	052 to 064	065	067, 069, 052	065, 059

7.2 Positioning the four types

The upper-right quadrant is almost empty in practice. A problem where each observation carries several labels and those labels are ordered — grading several independent quality criteria of one product, each on a five-point scale — has no single standard frame. It is usually handled as K independent ordinal problems, one per criterion, which is a multiclass-multioutput task rather than a multilabel one.

7.3 Routing a task to its metric

If the task is	And the concern is	Then steer on	Chapter
Binary, balanced	Overall correctness	Accuracy, F1	053, 059
Binary, imbalanced	Catching the rare event	Recall at fixed precision, PR-AUC	055, 064
Binary, imbalanced	Cost of the two error types	Expected cost at the tuned threshold	057, 062
Binary, ranking only	Quality of the ordering	ROC-AUC	063
Multiclass, balanced	Overall correctness	Accuracy, macro F1	053, 065
Multiclass, imbalanced	Rare classes not absorbed	Macro F1, balanced accuracy	065, 060
Multiclass, any	Which pairs get confused	Confusion matrix	052
Ordinal	Magnitude of the errors	Quadratic weighted kappa, MAE on ranks	067, 052
Ordinal	Concentration around the diagonal	Confusion matrix	052
Multilabel, output consumed whole	All labels right at once	Subset accuracy	065
Multilabel, output consumed per label	Individual decisions	Hamming loss plus per-label F1	059, 065
Multilabel, rare labels	Rare labels not ignored	Macro F1	065

7.4 Three invariants

Binary, multiclass and ordinal share the same shape of y. Only the number of modalities and the presence of an order separate them.
Multilabel is the only type whose target is a matrix, and the only one where a prediction can be partially correct.
The type of classification determines the admissible metrics before it determines the algorithms. Most algorithms handle several types; almost no metric does.

8. Reference sheet

The three qualification questions

Q1  Can one observation carry several labels at once?
        Yes  ->  MULTILABEL, target Y of shape (n, K)
        No   ->  Q2
Q2  How many mutually exclusive classes?
        K = 2  ->  BINARY, target y of shape (n,)
        K > 2  ->  Q3
Q3  Do the classes carry a natural total order?
        Yes  ->  ORDINAL,    y of shape (n,), ordered dtype
        No   ->  MULTICLASS, y of shape (n,), nominal

Boxed formulas

Binary decision rule
    f(x) = 1 if p(x) >= t, else 0
    where p(x) = P(Y = 1 | X = x) and t is a business choice, not 0.5 by right

One-vs-Rest        K classifiers,          predict = argmax_k s_k(x)
One-vs-One         K(K-1)/2 classifiers,   predict = majority vote

Cumulative decomposition, K levels
    q_k = P(y > c_k),  k = 1 .. K-1
    P(c_1)   = 1 - q_1
    P(c_k)   = q_(k-1) - q_k     for 1 < k < K
    P(c_K)   = q_(K-1)

Quadratic weighted kappa
    w_ij   = (i - j)^2 / (K - 1)^2
    E_ij   = row_i * col_j / n
    kappa  = 1 - sum(w * O) / sum(w * E)
    kappa = 1 perfect, 0 chance-level, negative worse than chance

Hamming loss       = wrong cells / (n * K)
Subset accuracy    = rows entirely correct / n

Reading values

Quantity	Value	Reading
`y.ndim`	1	Single-label: binary, multiclass or ordinal
`y.ndim`	2	Multilabel, or one-hot encoded multiclass
`Y.sum(axis=1)`	Constant 1	One-hot multiclass, almost certainly
`Y.sum(axis=1)`	Anything else	Multilabel
`y.dtype`	`category`, `ordered=True`	The analyst has declared an ordinal target
`model.coef_.shape`	`(1, p)`	Binary
`model.coef_.shape`	`(K, p)`, K > 2	Multiclass, multinomial
`predict_proba` row sum	Exactly 1	Single-label
`predict_proba` row sum	Anything else	Multilabel, K independent models
Quadratic kappa	≥ 0.80	Strong agreement on the ordinal scale
Quadratic kappa	0.60 to 0.80	Usable, check the confusion matrix
Quadratic kappa	≤ 0.40	The order is barely being captured
Hamming loss ≈ positive rate	—	Suspect the empty-set predictor

Conditions of use, and of non-use

Type	Use it when	Do not use it when
Binary	The downstream decision has two outcomes	The business needs graded output and you are grouping only for convenience
Multiclass	The classes are exclusive, exhaustive and unordered	An order exists and the cost of an error depends on its magnitude
Ordinal	The classes are ordered and no distance is defined	The levels come from binning a measured quantity you still have — regress on that instead (chapter 010)
Multilabel	An observation can carry 0, 1 or several labels	Exactly one label always applies — that is multiclass, and the softmax constraint is information

The scikit-learn calls that matter

python

# qualification aid, not a qualification
from sklearn.utils.multiclass import type_of_target
type_of_target(y)                      # binary | multiclass | multilabel-indicator

# binary: name the positive class everywhere
precision_score(y_true, y_pred, pos_label=1)
proba = model.predict_proba(X)[:, list(model.classes_).index(1)]

# multiclass: decomposition, only for intrinsically binary learners
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
OneVsRestClassifier(LinearSVC())       # K models,          linear-cost learners
OneVsOneClassifier(SVC())              # K(K-1)/2 models,   quadratic-cost learners

# multiclass: averaging is not optional
f1_score(y_true, y_pred, average="macro")     # equal weight per class
f1_score(y_true, y_pred, average="weighted")  # weight by support
f1_score(y_true, y_pred, average="micro")     # equals accuracy in single-label

# ordinal: declare the order, then use order-aware metrics
pd.CategoricalDtype(categories=[...], ordered=True)
cohen_kappa_score(y_true, y_pred, weights="quadratic")
mean_absolute_error(y_true_ranks, y_pred_ranks)

# multilabel: encode, fit, and read label by label
from sklearn.preprocessing import MultiLabelBinarizer
MultiLabelBinarizer(classes=[...])            # fix the column order explicitly
from sklearn.multioutput import MultiOutputClassifier, ClassifierChain
MultiOutputClassifier(base)                   # binary relevance
ClassifierChain(base, order=[...], cv=5)      # correlations, cv reduces exposure bias
hamming_loss(Y_true, Y_pred)
classification_report(Y_true, Y_pred, target_names=labels, zero_division=0)

# task, not family
from sklearn.base import is_classifier, is_regressor
is_classifier(LogisticRegression())    # True

Arguments whose defaults mislead

Call	Default	Why it misleads
`precision_score`, `recall_score`, `f1_score`	`pos_label=1`	Silently wrong if the class of interest is not labeled 1; raises on text labels
`LogisticRegression().classes_`	Ascending sort	Alphabetical order on text labels puts `"Fraud"` at index 0
`predict_proba(X)[:, 1]`	—	Returns the complementary probability under text labels
`cohen_kappa_score`	`weights=None`	Unweighted kappa is as blind to magnitude as accuracy
`accuracy_score` on a 2-D `Y`	—	Silently becomes subset accuracy, a much harsher metric
`MultiLabelBinarizer()`	`classes=None`	Column order inferred from data; unstable across runs and splits
`SVC`	OvO	Not documented at the call site; changes the model count from K to K(K−1)/2
`f1_score(average="micro")`	—	In single-label problems it equals accuracy exactly, which is rarely the intent

9. Common reasoning mistakes

MISTAKE — Calling a multilabel problem multiclass

The criterion is not the number of classes but the number of labels per observation. A three-class problem in which one observation can carry two labels is multilabel, not multiclass.

Correct formulation : "Multiclass assumes the classes are mutually exclusive; as soon as one observation can carry several labels at once, the problem is multilabel and the target becomes an (n, K) matrix."

MISTAKE — Treating an ordinal target as a regression without justification

Coding Low, Medium, High, Critical as 0, 1, 2, 3 and then regressing assumes the gaps between consecutive levels are equal. An ordinal scale does not carry that information (Stevens, 1946).

Correct formulation : "Regression on ranks is a pragmatic approximation whose equal-spacing assumption must be stated and, where possible, justified by the business; it is not the reference treatment."

MISTAKE — Treating an ordinal target as nominal multiclass

The nominal treatment makes every confusion equivalent. Confusing Critical with Low is then counted exactly like confusing High with Critical, while the business cost differs by an order of magnitude. Section 4.3 shows two predictions with identical accuracy, 0.90, and quadratic kappas of 0.952 and 0.571.

Correct formulation : "The nominal treatment of an ordinal target is acceptable as a starting baseline, provided accuracy is supplemented by a metric sensitive to the magnitude of the error, such as MAE on ranks or quadratic weighted kappa."

MISTAKE — Reaching for Cohen's kappa to handle the ordinal case

Unweighted kappa corrects for chance agreement, not for the magnitude of a confusion. In section 4.3 it returns 0.865 for both the one-rank error and the three-rank error, exactly as accuracy does. The correction that matters is the weighting, and it has to be requested.

Correct formulation : "Ordinal evaluation requires weights='linear' or weights='quadratic'; unweighted kappa is a nominal metric."

MISTAKE — Believing logistic regression is a regression model

The name refers to the generalized linear model family and to Verhulst's logistic function, not to the nature of the task. The target is categorical.

Correct formulation : "Logistic regression is a classification algorithm. It regresses the logit of a probability on the explanatory variables; the class decision comes from comparing that probability with a threshold."

MISTAKE — Letting the positive class be defined by alphabetical order

With text labels, scikit-learn's default order is alphabetical. "Fraud" becomes class 0 and "Normal" class 1, so predict_proba(X)[:, 1] returns the probability of being normal. Section 2.4 shows the same model reporting 0.7955 and 0.2045 for the same transaction depending only on how the target was coded.

Correct formulation : "The positive class is designated explicitly as the class of interest, usually the rare and costly event, and that designation accompanies every report of the metrics."

MISTAKE — Assuming K > 2 forces an OvR or OvO decomposition

Trees, forests, boosting methods, k-NN, Naive Bayes, multinomial logistic regression and neural networks all handle K > 2 natively. Decomposition concerns only intrinsically binary algorithms.

Correct formulation : "One-vs-Rest and One-vs-One are compatibility mechanisms for binary classifiers, applied transparently by the library, not a mandatory step in multiclass classification."

MISTAKE — Choosing OvR over OvO because it trains fewer models

Model count is not cost. Each OvO model sees roughly 2n/K rows instead of n. For a learner whose training cost is quadratic in n, OvO is cheaper overall despite training K(K − 1)/2 models. Section 3.3 measures 15 models finishing 2.8 times faster than 6.

Correct formulation : "The decomposition is chosen from the complexity of the base learner in n, not from the number of sub-models: OvR for linear-cost learners, OvO for quadratic-cost ones."

MISTAKE — Steering a multilabel model on subset accuracy alone

Subset accuracy requires all labels of an observation to be simultaneously correct. Across K labels it collapses mechanically as K grows, even for a model whose individual decisions are good.

Correct formulation : "A multilabel model is steered on a set of metrics: Hamming loss for the global view, macro F1 for sensitivity to rare labels, and per-label F1 for diagnosis."

MISTAKE — Steering a multilabel model on Hamming loss alone

The symmetric error. On a sparse label set, a model that always returns the empty set achieves a Hamming loss equal to the positive-cell rate. Section 5.5 measures 0.0339 on 40 labels — apparently 96.6 % of decisions correct — with F1 micro and macro both exactly 0.

Correct formulation : "Hamming loss is to multilabel what accuracy is to imbalanced binary classification: a majority-dominated metric that a trivial predictor optimizes. It is read alongside a metric that requires true positives."

MISTAKE — Trusting `type_of_target` to qualify the task

The function reports multiclass for both nominal and ordinal targets, and multilabel-indicator for both genuine multilabel matrices and one-hot encoded multiclass ones. Two of the three discriminating questions are outside its reach.

Correct formulation : "type_of_target checks the format of an array. The type of the task is asserted by the analyst from domain knowledge, and only then encoded into the data structures."

MISTAKE — Grouping classes for convenience and expecting to ungroup later

Collapsing a five-level satisfaction scale to two levels before training discards the granularity permanently. The model has never seen the intermediate levels and cannot restore them, whatever post-processing is applied.

Correct formulation : "Grouping classes is a specification decision taken with the consumer of the prediction; the finer formulation can always be coarsened afterwards at the presentation layer, and the reverse is impossible."

10. Summary

THE THREE QUALIFICATION QUESTIONS
    1. Several labels per observation ?    Yes   -> MULTILABEL
    2. How many exclusive classes ?        K = 2 -> BINARY
    3. Are the classes ordered ?           Yes   -> ORDINAL
                                           No    -> MULTICLASS
    Question 1 changes the SHAPE of the target. Questions 2 and 3 do not.

SHAPE OF THE TARGET
    Binary      y of shape (n,)      values 0 / 1
    Multiclass  y of shape (n,)      K unordered modalities
    Ordinal     y of shape (n,)      K ordered modalities, no distance
    Multilabel  Y of shape (n, K)    binary matrix, row sum unconstrained

FORMAL SIGNATURE OF MULTILABEL
    Y.sum(axis=1) constant and equal to 1   ->  encoded multiclass
    Y.sum(axis=1) anything else             ->  multilabel
    type_of_target cannot separate the two, nor ordinal from nominal.

POSITIVE CLASS, BINARY
    Always designated explicitly: the rare, costly, actionable event.
    Inversion swaps recall and specificity, turns AUC into 1 - AUC,
    and moves the PR-AUC baseline from one prevalence to the other.
    A recall of 0.92 means nothing until the positive class is named.

MULTICLASS DECOMPOSITION
    One-vs-Rest  K classifiers,          sub-problems imbalanced 1 against K-1
    One-vs-One   K(K-1)/2 classifiers,   sub-problems of size about 2n/K
    Choose on the COST of the base learner in n, not on the model count.
    Linear cost -> OvR.  Quadratic cost -> OvO, and it wins by 2.8x at K = 6.

ORDINAL TARGET
    As multiclass : the order is lost, every confusion counts the same.
    As regression : equal spacing assumed, unfounded (Stevens 1946).
    Conforming    : cumulative models, magnitude-sensitive metrics.
    Accuracy AND unweighted kappa are both blind to magnitude.
    Quadratic weighted kappa: 0.952 for a 1-rank error, 0.571 for a 3-rank one.

MULTILABEL
    The only type with a matrix target and partially correct predictions.
    Binary relevance ignores label correlations; classifier chains use them.
    Subset accuracy alone is too severe; Hamming loss alone is too generous.
    An empty-set predictor scores 0.0339 Hamming loss and 0.0 F1.

LOGISTIC REGRESSION
    Name inherited from the logistic function (Verhulst, 1838) and from the
    generalized linear model family (Nelder and Wedderburn, 1972).
    Task solved: CLASSIFICATION. Native output: a probability.
    is_classifier(LogisticRegression()) is True. Detail in chapter 036.

Summary statement

The type of a classification task is read from the shape and the declared structure of the target variable, never from the vocabulary in use: a vector with two modalities denotes a binary problem, a vector with K modalities a multiclass problem — ordinal if those modalities are ranked without being measured — and a binary matrix a multilabel problem; and the name of an algorithm, logistic regression foremost among them, records its mathematical family and never the task it solves.

Associated quizzes

011.1-quiz-task-qualification.md
011.2-quiz-binary-classification.md
011.3-quiz-multiclass-classification.md
011.4-quiz-ordinal-classification.md
011.5-quiz-multilabel-classification.md
011.6-quiz-logistic-regression.md
011.7-quiz-comparative-summary.md

Next chapter : 012.0-ml-project-lifecycle.md

1. Qualifying a classification task: three discriminating questions

1.1 What chapter 010 settled, and what it left open

1.2 The three questions, in order

1.3 Why the order of the questions is not arbitrary

1.4 The taxonomy at a glance

1.5 The type is a modeling decision, not an observation

1.6 What the library can and cannot tell you

2. Binary classification

2.1 The 0/1 convention and what it buys

2.2 Worked data — payment card fraud

2.3 The positive class and the consequences of choosing it

2.4 The alphabetical-order trap, measured

2.5 From score to class: the order of operations

2.6 What the choice is worth in dollars

3. Multiclass classification

3.1 Multiclass is not multilabel

3.2 Worked data — support ticket routing

3.3 Decomposition strategies for intrinsically binary algorithms

3.4 The ambiguity zones, measured

3.5 Natively multiclass algorithms

3.6 The shape of the native output, across the three single-label types

4. Ordinal classification

4.1 Worked data — internal credit risk grade

4.2 The two naive treatments and what each costs

4.3 Measuring the loss: accuracy is blind to magnitude

4.4 Quadratic weighted kappa, derived by hand

4.5 Approaches that respect the order

4.6 When regression on ranks is defensible

5. Multilabel classification

5.1 Worked data — photograph tagging

5.2 The strict distinction from one-hot multiclass

5.3 Decomposition: binary relevance and classifier chains

5.4 Metrics: subset accuracy and Hamming loss

5.5 The pathology: the model that predicts nothing

6. Logistic regression: a classifier with a misleading name

6.1 Where the name comes from

6.2 The mechanism, step by step

6.3 What scikit-learn says

6.4 Other names to watch

7. The four types side by side

7.1 Recap table

7.2 Positioning the four types

7.3 Routing a task to its metric

7.4 Three invariants

8. Reference sheet

9. Common reasoning mistakes

10. Summary