The clean readings you have used so far were prepared for you. The file NorthPeak actually collects, data/raw/readings_raw.csv, has missing temperatures, duplicated rows and a sensor placeholder of 999.0. Raw data has three common defects, and each one follows the same method: find it, decide what it means, then act. This lesson shows the three defects on the raw file and why the decision step needs the domain, not only the statistics.
A missing value is an empty cell. pandas shows it as NaN. A duplicate is a row that appears twice. An outlier is a value far from all the others.
For each defect the method is the same. First find it and count it. Then decide what it means. Then act. Never act before you count.
| Defect | Find | Typical decisions |
|---|---|---|
| Missing value | df.isna().sum() | Drop the row, or fill with a median, or leave it |
| Duplicate | df.duplicated().sum() | Drop the copies with drop_duplicates() |
| Outlier | describe(), a rule, or a fence | Keep it (real), or turn it into NaN (error) |
The hard part is the decision. An outlier can be a real hot day. It can also be a broken sensor. The number alone does not tell you. The domain does.
data/raw/readings_raw.csv is the dirty copy of the readings. isna().sum() shows 295 missing temperatures, 2 % of the rows. duplicated().sum() shows 150 duplicated rows.
For outliers, one common tool is the IQR fence. Take the first quartile Q1 and the third quartile Q3. The fence is Q3 plus 1.5 times the gap between them. On the raw temperatures: Q1 is 35.65, Q3 is 56.5, so the fence is 87.8 degrees. Thirty rows are above it.
Now look at those thirty. Fifteen are exactly 999.0. That is a placeholder: an old gateway writes 999 when the sensor is unplugged. The other fifteen are between 88 and 93 degrees, almost all in July and August, all on compressors. Those are real hot days.
The fence found both kinds. Only the domain separates them. The sensor guide in docs/manuals/ says a casing temperature above 120 is impossible. That rule flags exactly the fifteen placeholders and none of the hot days.
Placeholders poison averages. With the fifteen 999.0 rows, the mean raw temperature is 47.2. Without them it is 46.2. One degree, from fifteen rows out of fourteen thousand. The median does not move: 46.1 both ways. When mean and median disagree, look for a placeholder.