- Block 19 — Putting a model into production
- Objective
- treat a deployed model as an object on a decay curve and build the instrumentation
that measures it; define covariate shift, label shift and concept drift by what changed, simulate
all three on one frozen model and price the damage each does to accuracy, calibration and the
optimal threshold; correct a prior shift with no labels; separate the sudden, gradual, incremental
and recurring patterns; build the label-free monitoring stack and measure its lead over the
accuracy alarm; run Kolmogorov-Smirnov, PSI, Jensen-Shannon, Wasserstein and chi-squared on one
stream; measure the false-alarm rate of 200 uncorrected daily tests and repair it; use a domain
classifier to catch a change no marginal test can see; measure the survivorship bias in returned
labels and the holdout that removes it; tune an alert rule against the trade between detection
delay and false alarms; price the responses against the diagnoses; compare scheduled with
triggered retraining over eighty weeks; and write the monitoring specification down.
- Estimated duration
- 130 minutes
- Prerequisites
- chapters 026, 028, 030, 034, 050, 052, 061, 062, 063, 075, 076 and 081
- Associated quizzes
082.1-quiz-the-stationarity-assumption.md to
082.12-quiz-the-monitoring-specification.md