Duplicates, Inconsistencies and Data Quality

Block 5 — Data cleaning

Objective

hold an explicit data-quality framework with an executable test behind each dimension; detect and qualify duplicates, value inconsistencies, impossible values and mis-inferred types; run a reproducible quality audit that produces a control report; and understand why failure caused by poor data quality is silent in machine learning.

Estimated duration

50 minutes

Prerequisites

chapters 005 (dataset structure), 013 (exploratory analysis), 014 (descriptive statistics), 018 (missing values)

Associated quizzes

020.1-quiz-quality-dimensions.md to 020.7-quiz-garbage-in-garbage-out.md

Chapter 018 handled a single quality dimension — completeness — and chapter 019 handled what to do about it. This chapter handles the other five, and it handles them the same way: not as advice, but as checks that each return a number.