- Block 15 — The classification metrics
- Objective
- treat the decision threshold as a first-class deployed
artifact, fitted, validated, versioned, monitored and owned separately from
the model; measure what the default 0.5 costs and break the three conditions
that would justify it; derive
t* = c_FP / (c_FP + c_FN) and the prior-shift
correction and verify both by exhaustive sweep; compute six defensible answers
to "what threshold?" on one model; work TunedThresholdClassifierCV end to
end; measure the optimism of tuning and reporting on the same rows, and the
spread and flatness of the selected value; derive the two-threshold abstain
rule and cost its deferral queue; measure the gain and the governance hazard
of per-segment thresholds; watch a fixed threshold degrade under drift while
accuracy improves; compare the threshold against class weights and resampling;
and specify the record that makes a threshold auditable. - Estimated duration
- 110 minutes
- Prerequisites
- chapters 029, 030, 034, 050, 051, 052, 053, 054, 055,
057 and 061
- Associated quizzes
062.1-quiz-two-artifacts.md to
062.12-quiz-documenting-a-threshold.md