The lab kit of the course: https://github.com/hrhouma2/aiopsatlas-ml-data-diagnostics-labs-en
The gateway of NorthPeak, the small computer that collects the sensor values, exported one year of readings before anyone checked them. That file is data/raw/readings_raw.csv. Your job today is not to fix it. Your job is to find every defect and count it. Missing values, duplicated rows, impossible values. At the end you print one table with nine counts. Exercise 3 will use it as a checklist.
cd aiopsatlas-ml-data-diagnostics-labs-en
.\.venv\Scripts\Activate.ps1cd aiopsatlas-ml-data-diagnostics-labs-en
source .venv/bin/activateCreate a file week02/explore_raw.py. Add the code of each step to it. Run it with python week02/explore_raw.py after every step. Put these two lines at the top so wide tables print on one line:
import pandas as pd
pd.set_option("display.width", 120)
pd.set_option("display.max_columns", 12)data/raw/readings_raw.csv. The rows are shuffled. The label column fault_next_7d is not there: a gateway does not know the future.
reading_id,machine_id,date,load_pct,ambient_c,temperature_c,vibration_mm_s,pressure_bar,power_kw
14587,M040,2025-12-18,92.6,-7.3,43.8,5.63,4.81,60.0
795,M003,2025-03-06,45.7,1.1,42.4,4.97,6.61,23.2
14576,M040,2025-12-07,28.8,-5.8,22.6,4.01,3.82,22.3
6519,M018,2025-11-10,55.4,4.1,55.7,5.97,9.48,45.5
10063,M028,2025-07-27,31.3,23.4,45.3,2.72,1.23,8.5reading_id — a unique number per reading, 1 to 14,600 in the clean file.machine_id — the machine, M001 to M040.date — the day, one row per machine per day of 2025.load_pct — share of the rated capacity used, 0 to 100.ambient_c — air temperature of the room, in degrees.temperature_c — casing temperature, in degrees.vibration_mm_s — vibration speed, in millimetres per second.pressure_bar — process pressure, in bar.power_kw — electrical power drawn, in kilowatts.Load the file. Print its shape and its first rows.
raw = pd.read_csv("data/raw/readings_raw.csv")
print(raw.shape)
print(raw.head())Expected output (real, from the kit):
(14750, 9)
reading_id machine_id date load_pct ambient_c temperature_c vibration_mm_s pressure_bar power_kw
0 14587 M040 2025-12-18 92.6 -7.3 43.8 5.63 4.81 60.0
1 795 M003 2025-03-06 45.7 1.1 42.4 4.97 6.61 23.2
2 14576 M040 2025-12-07 28.8 -5.8 22.6 4.01 3.82 22.3
3 6519 M018 2025-11-10 55.4 4.1 55.7 5.97 9.48 45.5
4 10063 M028 2025-07-27 31.3 23.4 45.3 2.72 1.23 8.514,750 rows. The clean file has 14,600. Forty machines times 365 days is 14,600. So 150 rows are extra. Write that number down.
info() prints the number of filled values per column.
raw.info()RangeIndex: 14750 entries, 0 to 14749
Data columns (total 9 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 reading_id 14750 non-null int64
1 machine_id 14750 non-null str
2 date 14750 non-null str
3 load_pct 14750 non-null float64
4 ambient_c 14750 non-null float64
5 temperature_c 14455 non-null float64
6 vibration_mm_s 14529 non-null float64
7 pressure_bar 14601 non-null float64
8 power_kw 14750 non-null float64Three columns are below 14,750: temperature_c, vibration_mm_s, pressure_bar. Those have missing values.
isna() marks every empty cell. sum() counts them per column.
print(raw.isna().sum())reading_id 0
machine_id 0
date 0
load_pct 0
ambient_c 0
temperature_c 295
vibration_mm_s 221
pressure_bar 149
power_kw 0295 missing temperatures. That is 2 % of the rows. Keep this number; it will change in Step 5.
duplicated() marks a row when an identical row appeared earlier. Count them two ways: full rows, and reading_id alone.
print("full duplicates :", raw.duplicated().sum())
print("same reading_id :", raw.duplicated(subset=["reading_id"]).sum())
print(raw[raw["reading_id"] == 100])full duplicates : 150
same reading_id : 150
reading_id machine_id date load_pct ambient_c temperature_c vibration_mm_s pressure_bar power_kw
245 100 M001 2025-04-10 67.6 6.6 47.3 4.18 7.24 30.4
8442 100 M001 2025-04-10 67.6 6.6 47.3 4.18 7.24 30.4150, both ways. The gateway resent 150 readings after a network retry. Reading 100 is one of them. Both copies are identical, so keeping one loses nothing.
drop_duplicates() keeps the first copy of each row. Then count the missing values again.
readings = raw.drop_duplicates()
print(readings.shape)
print(readings.isna().sum())(14600, 9)
reading_id 0
machine_id 0
date 0
load_pct 0
ambient_c 0
temperature_c 291
vibration_mm_s 219
pressure_bar 146
power_kw 014,600 rows: the right count. The missing temperatures went from 295 to 291. Four duplicated rows had an empty temperature, so they were counted twice. Always remove duplicates before you count anything else.
describe() shows min and max. Read them with the sensor guide in mind (docs/manuals/05-sensor-calibration-and-data-quality.md in the kit): load 0 to 100, temperature below 120, vibration below 30.
cols = ["load_pct", "temperature_c", "vibration_mm_s", "pressure_bar"]
print(readings[cols].describe().round(2)) load_pct temperature_c vibration_mm_s pressure_bar
count 14600.00 14309.00 14381.00 14454.00
mean 53.78 47.19 4.24 5.40
std 20.34 34.14 2.77 2.77
min -99.40 3.00 0.10 0.26
25% 38.00 35.60 3.23 3.43
50% 53.00 46.00 4.02 5.56
75% 68.80 56.50 5.03 8.00
max 100.00 999.00 148.00 10.45Three lines are wrong. load_pct has a minimum of -99.4: a load cannot be negative. temperature_c has a maximum of 999.0: a placeholder. vibration_mm_s has a maximum of 148.0: thirty times the normal level. pressure_bar looks fine.
Write one rule per problem and count the rows it flags. Then print the negative loads.
n_999 = (readings["temperature_c"] == 999.0).sum()
n_neg = (readings["load_pct"] < 0).sum()
n_spike = (readings["vibration_mm_s"] > 30).sum()
print("temperature == 999.0 :", n_999)
print("load_pct < 0 :", n_neg)
print("vibration > 30 :", n_spike)
print(readings.loc[readings["load_pct"] < 0, ["reading_id", "machine_id", "date", "load_pct"]].sort_values("reading_id"))temperature == 999.0 : 15
load_pct < 0 : 12
vibration > 30 : 10
reading_id machine_id date load_pct
11924 690 M002 2025-11-21 -42.6
4969 783 M003 2025-02-22 -34.6
12839 1914 M006 2025-03-30 -27.6
...
13679 9912 M028 2025-02-26 -99.4
12038 10755 M030 2025-06-19 -83.2
4577 11098 M031 2025-05-28 -68.9Fifteen placeholders, twelve negative loads, ten vibration spikes. Look at the loads: -42.6, -34.6, -27.6. The sizes are normal. Only the sign is wrong.
Put the nine counts in one Series and print it. This is your deliverable.
defects = pd.Series(
{
"rows in the raw file": len(raw),
"duplicated rows": int(raw.duplicated().sum()),
"rows after drop_duplicates": len(readings),
"missing temperature_c": int(readings["temperature_c"].isna().sum()),
"missing vibration_mm_s": int(readings["vibration_mm_s"].isna().sum()),
"missing pressure_bar": int(readings["pressure_bar"].isna().sum()),
"temperature_c == 999.0": int(n_999),
"load_pct < 0": int(n_neg),
"vibration_mm_s > 30": int(n_spike),
},
name="count",
)
print(defects)rows in the raw file 14750
duplicated rows 150
rows after drop_duplicates 14600
missing temperature_c 291
missing vibration_mm_s 219
missing pressure_bar 146
temperature_c == 999.0 15
load_pct < 0 12
vibration_mm_s > 30 10You changed nothing in the file. You only counted. That is the right first step.
describe() looked fine, and which three looked wrong?(readings["temperature_c"] > 120).sum() return, and why?duplicated().sum() found exactly 150 copies.drop_duplicates().pressure_bar looked fine. load_pct (min -99.4), temperature_c (max 999.0) and vibration_mm_s (max 148.0) looked wrong.999.0 placeholders. The real maximum is 93.0.Open data/raw/machines_raw.csv and data/raw/incidents_raw.csv. Use value_counts() on machine_type and site. Use str.contains("/") on the incident date column. Use isna().sum() on technician. Count the defects of those two files. You should find eight spellings of the machine type, one lowercase site, 23 dates written day first, and 8 blank technicians.
week02/exercise_1_solution.py in the kit"""Week 2, Exercise 1 - Explore the raw readings.
Run from the kit root, with the venv active:
python week02/exercise_1_solution.py
Finds every defect of data/raw/readings_raw.csv without fixing anything:
1. row count and shape;
2. info(): which columns have missing values;
3. isna().sum(): how many missing values per column;
4. duplicated(): the 150 rows that appear twice;
5. drop_duplicates(), then count the missing values again;
6. describe(): impossible minimums and maximums;
7. the 999.0 temperatures, the negative loads, the vibration spikes;
8. one defect table, the deliverable.
"""
import pandas as pd
pd.set_option("display.width", 120)
pd.set_option("display.max_columns", 12)
# Step 1 - load and count
print("== Step 1: load and count ==")
raw = pd.read_csv("data/raw/readings_raw.csv")
print(raw.shape)
print(raw.head())
print()
# Step 2 - info
print("== Step 2: info ==")
raw.info()
print()
# Step 3 - missing values per column
print("== Step 3: missing values per column ==")
print(raw.isna().sum())
print()
# Step 4 - duplicated rows
print("== Step 4: duplicated rows ==")
print("full duplicates :", raw.duplicated().sum())
print("same reading_id :", raw.duplicated(subset=["reading_id"]).sum())
pair = raw[raw["reading_id"] == 100]
print(pair)
print()
# Step 5 - drop the duplicates, count again
print("== Step 5: drop duplicates, count again ==")
readings = raw.drop_duplicates()
print(readings.shape)
print(readings.isna().sum())
print()
# Step 6 - describe: impossible values
print("== Step 6: describe ==")
cols = ["load_pct", "temperature_c", "vibration_mm_s", "pressure_bar"]
print(readings[cols].describe().round(2))
print()
# Step 7 - count each kind of impossible value
print("== Step 7: impossible values ==")
n_999 = (readings["temperature_c"] == 999.0).sum()
n_neg = (readings["load_pct"] < 0).sum()
n_spike = (readings["vibration_mm_s"] > 30).sum()
print("temperature == 999.0 :", n_999)
print("load_pct < 0 :", n_neg)
print("vibration > 30 :", n_spike)
print(readings.loc[readings["load_pct"] < 0, ["reading_id", "machine_id", "date", "load_pct"]].sort_values("reading_id"))
print()
# Step 8 - the defect table
print("== Step 8: defect table ==")
defects = pd.Series(
{
"rows in the raw file": len(raw),
"duplicated rows": int(raw.duplicated().sum()),
"rows after drop_duplicates": len(readings),
"missing temperature_c": int(readings["temperature_c"].isna().sum()),
"missing vibration_mm_s": int(readings["vibration_mm_s"].isna().sum()),
"missing pressure_bar": int(readings["pressure_bar"].isna().sum()),
"temperature_c == 999.0": int(n_999),
"load_pct < 0": int(n_neg),
"vibration_mm_s > 30": int(n_spike),
},
name="count",
)
print(defects)Run it with python week02/exercise_1_solution.py. The last block of its output is the defect table shown in Step 8.
All systems — FileNotFoundError: data/raw/readings_raw.csv. You are not in the kit root, or python data/make_dataset.py was never run. Type cd .. until ls (or dir) shows data and week02.
All systems — ModuleNotFoundError: No module named 'pandas'. The venv is not active. Run the activation command of the setup block. The prompt must start with (.venv).
All systems — the table prints on several lines with a \ at the end. The two pd.set_option lines are missing at the top of your script.
All systems — raw.isna().sum() prints 291, not 295. You already ran drop_duplicates() on raw. Step 3 counts on the raw file; Step 5 counts after the drop.
Windows only — Activate.ps1 cannot be loaded because running scripts is disabled. Run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned once, then activate again.
Linux and macOS only — python: command not found. Use python3 to create the venv. Once the venv is active, python works.