You rarely read all 14,600 rows of the NorthPeak readings; you look at a sample. When the sample does not look like the whole file, every number you compute from it is off, and the file is sorted in a way that makes this easy. A validation rule is the second safeguard: a range every value must respect before you trust it. This lesson shows a biased sample, a random one, and the six ranges from the sensor guide.
You rarely look at all the rows. You look at a sample: the first thousand, one month, one site. A sample is fine when it looks like the whole. It is a sampling bias when it does not.
The most common bias is the easiest to make. You take the first rows of the file. Files are sorted. The first rows all come from the same place or the same time.
The cure is a random sample: df.sample(1000, random_state=42). The random_state makes it the same sample every run. Then compare the sample to the whole. If a number differs a lot, the sample is biased.
readings.csv is sorted by machine, then by date. readings.head(1000) contains only M001, M002 and part of M003. Three pumps out of forty machines. The fault rate in that sample is 6.3 %. In the whole file it is 7.4 %.
readings.sample(1000, random_state=42) contains all 40 machines. Its fault rate is 8.4 %. Closer, and it covers every machine.
Time is a bias too. The mean temperature in January is 32.6 degrees. In July it is 60.5. A model trained on winter only would call every summer day a fault.
A validation rule is a range a value must respect. The sensor guide, docs/manuals/05-sensor-calibration-and-data-quality.md, gives one per column.
| Column | Valid range |
|---|---|
load_pct | 0 to 100 |
ambient_c | -30 to 45 |
temperature_c | -20 to 120 |
vibration_mm_s | 0.1 to 30 |
pressure_bar | 0.2 to 15 |
power_kw | 0.5 to 120 |
In Python a rule is one line: bad = (df["load_pct"] < 0) | (df["load_pct"] > 100). On the raw readings, bad.sum() gives 12. Twelve rows have a negative load. The same rule on vibration_mm_s gives 10, and on temperature_c gives 15.
A rule tells you a value is impossible. It does not tell you what the right value was. A negative load may be a sign error, so the absolute value fixes it. A 999.0 temperature holds no information, so nothing fixes it. Read the defect before you pick the fix.