- Block 15 — The classification metrics
- Objective
- define accuracy and its complement the error rate from the
confusion matrix and make explicit the two assumptions the formula hides — that
every error costs the same and that every observation weighs the same;
establish accuracy as a Bernoulli proportion, derive its binomial standard
error and build its confidence interval three ways (Wald, Wilson,
Clopper-Pearson) with an exact coverage comparison; compute the sample size
needed to distinguish two accuracies and answer "is 94.2 percent better than
93.8 percent"; establish the baselines any accuracy must beat and show that
accuracy has no meaningful zero of its own; measure the accuracy paradox under
imbalance; define balanced accuracy and its
adjusted=True rescaling; prove
and verify the micro-average identity in multiclass and define top-k accuracy;
define Cohen's kappa and exhibit a low kappa beside a high accuracy; introduce
MCC catching a degenerate classifier that accuracy praises; decompose the error
rate by cell, class and cost; and state the conditions under which accuracy is
the right metric. - Estimated duration
- 75 minutes
- Prerequisites
- chapters 011, 026, 027, 029, 030, 034, 050, 051 and 052
- Associated quizzes
053.1-quiz-definitions-and-assumptions.md to
053.9-quiz-reporting-accuracy.md