Consider the following sentence, representative of the applied literature:
"The dataset contains 10,000 instances described by 14 attributes, one of which is a categorical response variable. Each sample was annotated manually."
Here is the same content, restated without losing any information:
"The table has 10,000 rows and 14 columns. One column holds the value to be predicted, and that value is a category. Human operators filled that column in."
The two statements are strictly equivalent. The gap between them comes from inherited terminology. The field aggregates vocabulary from inferential statistics, computer science, relational databases, and direct borrowing from English. Several competing names therefore exist for the same object.
Point of caution. The ambiguity is not merely lexical. The word sample denotes two different objects depending on the discipline (section 2.1), and the word feature denotes sometimes any column, sometimes a column actually submitted to the model (section 3.3). Both confusions produce real methodological errors, not just clumsy sentences.
Rigorous definition
A structured collection of observations, each observation being described by a common set of variables measured or recorded under a homogeneous protocol.
Three elements of that definition are binding. The organization into rows and columns is explicit and stable. All observations are described by the same variables, which is what makes term-by-term comparison possible. A given variable keeps the same meaning and the same unit from one observation to the next.
Canonical form
The dataset is represented as a rectangular matrix in which each row is an observation and each column a variable. This form is called tabular data, or tidy data in Wickham's terminology (2014), which states three rules: one variable per column, one observation per row, one observational unit per table.
In plain terms
A table in which every row describes one observed case and every column one piece of information recorded about that case — the same piece of information for every case.
Point of caution
A pile of heterogeneous files is not a dataset in the sense above. It becomes one after consolidation, and consolidation is a full engineering step in its own right, with its own decisions and its own failure modes.
| Symbol | Name | What it counts | Synonyms you will meet |
|---|---|---|---|
| n | Number of observations | The rows | Sample size, number of instances, number of examples |
| p | Number of variables | The columns | Number of attributes, number of features, dimensionality |
| X | Feature matrix | n rows × p columns | Design matrix |
| y | Target vector | n values | Response vector, labels |
A dataset described as "n = 10,000, p = 14" reads: 10,000 observations, each described by 14 variables.
Point of caution. No convention here is universal. The statistical literature writes the number of variables p, for predictors; the machine learning literature sometimes writes d or m. Worse, depending on the author, p is either the total number of columns or only the number of explanatory variables. Ambiguity is removed by stating what you mean, never by assuming a convention.
Rigorous definition
The number of variables describing each observation — equivalently, the dimension of the vector space in which each observation is represented as a point.
Geometric reading
A dataset with p variables places every observation at a point in a p-dimensional space. Two observations close together in that space have similar values across all variables. Most algorithms exploit that structure: explicitly for distance-based methods, implicitly for methods that partition the space.
Methodological consequence
The volume of the space grows exponentially with p. Holding n fixed, increasing p spreads the observations thinner and degrades any local estimate. That phenomenon is the curse of dimensionality, covered in chapter 025.
In plain terms
The number of columns describing each case. The higher it is, the more rows you need for induction to rest on anything.
The logical structure — an n × p matrix — is independent of the storage medium. The formats below all encode the same object.
| Format | Technical nature | Column typing | Suitable volume | Characteristic use |
|---|---|---|---|---|
| CSV | Delimited text file | None: everything is a string, types are inferred at read time | Up to a few hundred MB | Interchange between applications, exports |
| Spreadsheet (XLSX, ODS) | Binary file organized in sheets | Per cell, heterogeneous, unconstrained | A few tens of thousands of rows | Manual entry, business review |
| Relational table | Persisted SQL table | Declared and enforced by the schema | Millions to billions of rows | Production information systems |
| DataFrame | In-memory structure (pandas, polars, R) | Per column, homogeneous, explicit | Bounded by available memory | Analysis and modeling |
| Parquet | Column-oriented binary file | Declared in the metadata | Very large, with compression and partial reads | Analytical storage, data pipelines |
Point of caution on CSV. The absence of typing is the single most common cause
of badly inferred types. An identifier with leading zeros (00742) is read as an
integer and loses them. A decimal written with a comma separator is read as text.
A single N/A string blocks numeric inference for an entire column. These defects
are surfaced by df.info() (section 5.3), which is why that call comes early in
the protocol.
The same text can be printed, typed, displayed on a screen, or dictated. The medium determines the cost of handling it and the ways it can be damaged. It does not determine the text.
A dataset behaves the same way. Its logical structure remains an n × p matrix whether the medium is a 3 KB text file or a 400 GB distributed table. The practical consequence is that changing format never fixes a content problem: converting a badly populated CSV to Parquet produces a badly populated Parquet.
The number of observations means nothing in isolation. It is read relative to the number of variables and to the complexity of the phenomenon. The orders of magnitude below apply to supervised tabular modeling at moderate p, on the order of ten to fifty variables.
| Volume (n) | Verdict | Methodological consequence |
|---|---|---|
| Under 100 | Too small for grounded induction | Descriptive statistics and domain expertise. Any performance estimate is dominated by sampling noise |
| 100 to 1,000 | Usable under strict conditions | Simple, regularized models and few variables. Cross-validation is mandatory, since an isolated test set would be too small to be informative |
| 1,000 to 100,000 | The nominal range for tabular ML | Train / validation / test splitting is practical, ensemble methods are relevant, hyperparameter search is affordable |
| Over 100,000 | Comfortable | High-capacity models become viable. The binding constraint moves from volume to quality, representativeness, and compute cost |
Guiding principle. The ratio n / p matters more than n alone. Five thousand observations described by 8 variables is comfortable; the same 5,000 observations described by 3,000 variables is a high-dimensional problem and calls for entirely different methods.
Point of caution. In classification, the volume that governs what you can learn is not n but the count of the least represented class. A dataset of 200,000 observations of which 60 are positive is a 60-observation problem. This is developed in chapter 050.
Rigorous definition
The elementary unit of the dataset, corresponding to one statistical individual on which the full set of variables was recorded. An observation is materialized as one row of the n × p matrix.
Origin of the term
Statistics. The word individual there denotes the observed unit, whether it is a person, an object, an event, or a period of time.
In plain terms
One observed case, described completely.
Recommendation
Of the eight terms listed in section 2.2, this is the most neutral and the most precise. It is the reference term throughout this course.
Rigorous definition
A particular occurrence described by the vector of its attribute values, in the terminology of machine learning and knowledge representation.
Origin of the term
Computer science, by analogy with instantiating a class: the dataset schema plays the role of the class, each row the role of an instance.
Point of caution
The term is occasionally used in a narrower sense — an observation with no label, presented to the model at prediction time. That usage is a minority one, but it is attested, and it will confuse you if you are not expecting it.
Statistical sense — the historical one
A subset of a population, selected under a sampling design, from which properties of the whole population are inferred. A sample here is a set of observations, of size n.
"The sample contains 10,000 customers."
Machine learning sense — the library convention
Sample as used in learning libraries denotes a single row. The scikit-learn
documentation systematically defines X as an array of shape
(n_samples, n_features), where each sample is one individual observation.
"The dataset contains 10,000 samples."
Consequence
The same word denotes the whole set in one sentence and a single element in the other. Nothing but context — and often nothing but grammatical number — disambiguates it.
Recommendation
In professional writing, reserve sample for the statistical sense and use observation for the row. When reading, never assume which sense is meant.
Rigorous definition
A structured set of fields forming a unit of storage and access in a data management system. In the relational model, a record corresponds to a tuple of a relation, that is, a row of a table.
In plain terms
A file card: the unit the system reads, writes, or deletes in one piece.
Point of caution
A record in the sense of the information system does not necessarily coincide with an observation in the sense of modeling. A transaction table holds one record per transaction; if the chosen unit of analysis is the customer, one observation aggregates several records. That mismatch is the subject of section 2.3.
Rigorous definition
The representation of an observation as a point in a p-dimensional space, each coordinate being the value of one variable.
Origin of the term
The geometric vocabulary of statistical learning, used when the reasoning concerns distances, neighborhoods, or decision boundaries.
Point of caution
In loose usage, data point sometimes denotes a single isolated value — a cell rather than a row. That usage is improper in a technical context and should not be copied.
| Term | Discipline of origin | What it denotes | Characteristic context |
|---|---|---|---|
| Observation | Statistics | One row | Rigorous writing, methodological papers |
| Individual | Statistics | One row | Descriptive statistics, classical data analysis |
| Instance | Computer science, symbolic AI | One row | Machine learning literature |
| Sample | English, ML libraries | One row (ML sense) or a set of rows (statistical sense) | scikit-learn documentation, survey design |
| Record | Databases | One table row | Data engineering, SQL |
| Row | Tabular representation | One row | File and DataFrame manipulation |
| Example | Supervised learning | One labeled row | Descriptions of the training set |
| Data point | Geometry of learning | One row seen as a point | Reasoning about distances and neighborhoods |
All eight terms denote the same object in the context of this course. Their coexistence is a fact about the literature, not a substantive distinction. The only genuine ambiguity is sample.
What a row represents is not a given property of the dataset. It is a modeling decision, made upstream, and it determines the question the model will be able to answer.
Rigorous definition
The reference entity to which the measurement applies and to which the inference refers. It defines what n counts and it bounds the scope of any conclusion drawn from the model.
Operational formulation
The unit of analysis answers the question: what is a prediction about? The model will produce one value per unit of analysis, no more and no less.
In plain terms
What one row represents, exactly.
Point of caution
The unit of analysis must coincide with the unit of business decision. A model whose unit of analysis is the transaction does not directly answer the question "is this customer at risk". A divergence forces an explicit aggregation step, with assumptions of its own that must be written down.
| Problem | One observation is | Order of magnitude of n |
|---|---|---|
| Predict subscription churn | One customer, at a given observation date | Number of active customers |
| Predict the sale price of a home | One completed property transaction | Number of historical sales |
| Detect a fraudulent transaction | One card transaction | Number of transactions processed |
| Predict equipment failure | One piece of equipment over a time window | Equipment count × number of windows |
| Classify an email as spam | One message | Number of messages in the corpus |
| Forecast product demand | One product × day pair | Number of products × number of days |
| Diagnose a condition from imaging | One examination of one patient | Number of examinations |
| Predict a campaign's conversion rate | One campaign | Number of campaigns run |
A remark on the last row. The number of campaigns a company has run is often counted in the dozens. The unit of analysis therefore determines the volume regime of section 1.5 directly, and with it the feasibility of the project. Moving from the campaign to the individual contact takes n from 40 to 400,000. That is not the same study, and it is not the same question.
Situation
An orders dataset has one row per order. A customer who placed several orders occupies several rows.
| order_id | customer_id | order_date | amount | channel | returned |
|---|---|---|---|---|---|
| ORD-1001 | C-042 | 2025-01-14 | 89.90 | web | 0 |
| ORD-1002 | C-317 | 2025-01-14 | 240.00 | store | 0 |
| ORD-1003 | C-042 | 2025-02-02 | 55.00 | web | 1 |
| ORD-1004 | C-042 | 2025-02-19 | 132.40 | web | 1 |
| ORD-1005 | C-988 | 2025-02-20 | 17.50 | web | 0 |
| ORD-1006 | C-317 | 2025-03-03 | 310.00 | store | 0 |
Consequence for splitting
A random split will, with high probability, place some of C-042's orders in the training set and others in the test set. The model is then evaluated on a customer whose idiosyncrasies it has already learned: preferred channel, typical basket size, propensity to return. The measured performance includes a component of memorizing already-seen entities, and it overstates real performance on unknown customers — which is the only performance that matters if the intended use concerns new customers.
Methodological answer
The split must be performed by group: every row belonging to the same customer
goes to the same side of the partition. The corresponding mechanism is
GroupKFold, covered in chapter 027, with customer_id as the grouping key.
Point of caution
This constraint gives customer_id a dual status. The column must be excluded
from the feature set (section 3.4) while being kept in the dataset to serve
as the grouping key. Dropping it too early makes a correct split impossible.
Related cases
A patient with several consultations, a machine with several readings, a branch with several months of activity, an image scored by several annotators.
A logistics manager reports handling 12,000 units last month. The statement is unusable until the unit is named: 12,000 items, 12,000 parcels, 12,000 pallets, and 12,000 orders describe activity levels separated by orders of magnitude.
Announcing that "the dataset has 50,000 rows" without saying what a row is has exactly the same defect. "50,000 what?" is the first question to ask, and the answer determines the volume actually available, the splitting strategy, and the scope of every conclusion you will draw.
The question to settle before touching anything:
What does one row of this dataset represent, exactly, and can a single real-world entity appear on more than one row?
The answer determines the effective n, the splitting strategy (chapters 026 and 027), the presence of entity-level leakage (chapter 028), and the business meaning of every prediction.
Rigorous definition
A quantity able to take different values depending on the observation considered, forming one dimension of description shared by all observations.
Origin of the term
Statistics. The term stands in opposition to constant: a column whose values are all identical is not a variable in the full sense.
In plain terms
A column — one piece of information recorded the same way for every case.
Point of caution
A zero-variance column is formally a column but has no discriminating power whatsoever. Detecting such columns is part of the initial inspection.
Attribute — Computer science and data mining. A descriptive property of an instance. The dominant term in the data mining literature and in academic benchmark datasets.
Field — Databases. A named, typed component of a record. The term is inseparable from the notion of schema: a field has a name, a declared type, and integrity constraints.
Dimension — Geometry of learning. An axis of the representation space. A false friend: in dimensional modeling for business intelligence, a dimension is a table of analysis axes, an entirely distinct sense.
Feature — The dominant term in practice. In its strict usage it denotes a variable actually submitted to the model as input, after selection and transformation. That strict usage is the one adopted here.
Point of caution
Variable and column are descriptive terms: they say what is present in the file. Feature is a functional term: it says what role the modeler has assigned. Confusing the two is the subject of section 3.3.
| Term | Discipline of origin | Its own nuance | Register |
|---|---|---|---|
| Variable | Statistics | Neutral, descriptive | Rigorous writing |
| Attribute | Data mining, AI | A property of an instance | Academic literature |
| Field | Databases | A typed component of a schema | Data engineering |
| Column | Tabular representation | Purely structural | File manipulation |
| Dimension | Geometry, BI | An axis of the representation space | Geometric reasoning |
| Feature | Professional practice | A variable submitted to the model | Everyday usage |
| Predictor, regressor | Inferential statistics | An explanatory variable of a model | Econometrics, biostatistics |
| Covariate | Experimental statistics | A variable adjusted for | Clinical trials, causal inference |
| Role | Identification criterion | Treatment | Reference |
|---|---|---|---|
| Identifier | Unique or near-unique value per entity, with no intrinsic business meaning | Exclude from the features, keep for traceability and grouping | Section 3.4, chapter 027 |
| Feature | Available and populated at the moment the prediction must be produced | Submit to the model, after encoding and transformation | Chapters 006, 022, 024 |
| Target | The quantity the model must estimate | Isolate in y, never leave it in X | Chapter 006 |
| Leakage variable | Populated after or because of the event to be predicted | Exclude without exception | Chapter 028 |
A file with 14 columns does not give you 14 features. It gives you 14 columns whose role remains to be determined, one by one. On an extract from a telecom churn file:
| Column | Role | Justification |
|---|---|---|
customer_id | Identifier | Designates the entity, carries no explanatory content |
tenure_months | Feature | Known before the event, carries information |
plan_type | Feature | Known before the event |
support_calls | Feature | Known before the event |
churned | Target | The quantity to be predicted |
churn_date | Leakage variable | Populated only if churn actually occurred |
churn_reason | Leakage variable | Collected at the moment of cancellation |
retention_agent_assigned | Leakage variable | Assigned in reaction to an announced cancellation |
Out of eight columns, only three are features. If kept, the three leakage columns would produce a model with near-perfect validation performance and no value whatsoever in production: they encode the answer. The mechanism is covered in chapter 028.
A single operational test, applicable to any column:
Is this value known and populated at the precise instant when the model must produce its prediction?
A negative answer disqualifies the column, whatever its correlation with the target. A high correlation is in fact the most common symptom of leakage, not evidence of quality.
Rigorous definition
An attribute whose function is to designate one entity unambiguously within a collection, independently of that entity's properties. In the relational model it corresponds to a primary key.
Characteristic property
The value of an identifier is arbitrary by construction. It results from an assignment convention — a sequence, a timestamp, a random draw — and not from any measurement of the phenomenon under study.
In plain terms
A case number: it lets you retrieve the file, it says nothing about its contents.
An identifier must be excluded from the features for three reasons, which compound.
No informational content. The value measures nothing. No regularity linking the identifier to the target can have any causal grounding.
Spurious correlation. Assignment frequently follows chronological order: low identifiers designate the oldest entities. A model then detects a statistically real relationship between the identifier and the target, which is nothing but a reflection of age. The relationship is captured through an arbitrary proxy instead of through the variable that actually carries it — which means the model degrades the moment the numbering scheme changes.
Memorization. A near-unique identifier lets a high-capacity model isolate every observation in its own leaf. The model memorizes the training set instead of inducing a rule. That is overfitting, covered in chapter 031.
Some identifiers are not arbitrary: their structure encodes genuine business information.
| Identifier | Encoded information | Correct treatment |
|---|---|---|
NUM-2019-BOS-00471 | Year of creation, originating branch | Extract creation_year and branch |
Serial number AX7-2021W34-0912 | Product line, week of manufacture | Extract product_line and build_week |
Product reference EL-TV-55-OLED | Department, category, format | Extract department, category, screen_size |
| IBAN | Country, banking institution | Extract country and bank_code |
Principle
The information is extracted into explicit, typed variables; the raw identifier is then excluded.
Why this is better than using the identifier directly
The extracted information is interpretable: a branch variable can be
analyzed, a substring cannot. It is robust: a change in numbering format
invalidates direct use of the identifier, not the extracted variable. And it is
auditable: the extraction rule is explicit and can be reviewed.
Point of caution
Keeping the raw identifier in addition to the extracted variables reintroduces all three risks above. Extraction replaces the identifier; it is not added alongside it.
The type of a variable determines which descriptive statistics are legitimate, which transformations apply, and which encoding is required. Classifying every column is therefore a precondition to any modeling.
Rigorous definition
A quantitative variable that can take, at least in principle, any value in an interval of the reals. Between any two observed values there is always an admissible intermediate value.
Recognition test
Does an intermediate value between two observed values make sense? Between 1.72 m and 1.73 m, the value 1.7248 m is admissible: the variable is continuous.
Examples
Price, salary, temperature, weight, height, duration, distance, concentration.
Consequence for treatment
Usable directly by most algorithms. Sensitive to scale for distance-based methods and for gradient descent, which calls for normalization or standardization (chapter 023). Tree-based methods are insensitive to it.
Point of caution
Measurement is always rounded to the precision of the instrument. Continuity is a property of the phenomenon, not of the recording.
Rigorous definition
A quantitative variable whose set of admissible values is countable, generally arising from a count.
Recognition test
Does the value come from counting, with intermediate values meaningless? There is no such thing as a home with 2.5 bedrooms.
Examples
Number of bedrooms, of children, of orders, of support calls, of insurance claims.
Consequence for treatment
Treated as numeric in the large majority of cases. A count variable with very low cardinality can also be treated as ordinal; the choice is settled by experimentation, not by principle.
Point of caution
Count variables are frequently strongly right-skewed with a mass at zero. A
log(1 + x) transformation is often beneficial for linear models.
Rigorous definition
A qualitative variable whose levels form exhaustive and mutually exclusive classes with no ordering relation between them.
Recognition test
Does ranking the levels from lowest to highest mean anything? If not, the variable is nominal: Boston is neither greater nor smaller than Denver.
Examples
City, country, brand, color, industry sector, contract type, acquisition channel, blood group.
Consequence for treatment
Encoding is mandatory before submission to an algorithm. One-hot encoding is the reference for linear models; it is expensive at high cardinality. Ordinal encoding is prohibited, because it introduces an ordering that does not exist (chapter 022).
Point of caution
Cardinality is the deciding factor. A variable with five levels and a variable with fifteen thousand levels call for different treatments entirely.
Rigorous definition
A qualitative variable whose levels carry a total ordering, without the gap between consecutive levels being quantifiable or assumed constant.
Recognition test
Does ranking make sense while the difference between two consecutive levels is not measurable? "Dissatisfied < Neutral < Satisfied" can be ranked, but the two gaps are not numerically comparable.
Examples
Satisfaction level, education level, energy rating, product tier, disease stage, credit rating.
Consequence for treatment
Ordinal encoding respecting the business order, with the level-to-value mapping defined explicitly and never inferred from alphabetical order.
Point of caution
Computing a mean over an ordinal encoding assumes the levels are equidistant — an assumption the very definition of ordinality rules out. A mean satisfaction of 2.7 is a management convenience, not a grounded statistical quantity.
Rigorous definition
A categorical variable with exactly two levels. It is nominal or ordinal depending on the domain, though the distinction has no practical effect at two levels.
Recognition test
Does the variable admit exactly two possible values, missing values aside?
Examples
Active customer, subscribed to an add-on, presence of an elevator, result of a test.
Consequence for treatment
Direct 0 / 1 encoding, with no one-hot expansion. The chosen convention must be documented: 1 conventionally denotes the level of interest.
Point of caution
When a binary variable is the target of a classification, the coding of the positive class determines how every asymmetric metric is read — precision, recall, PR-AUC. Inverting the convention invalidates the reading of the results (chapters 053 and 064).
Rigorous definition
A variable whose values designate instants or intervals on a chronological axis, equipped with an ordering relation and a metric of difference.
Recognition test
Does the value designate an instant, and does the difference between two values mean something in units of time?
Examples
Subscription date, transaction timestamp, date of birth, date of last login.
Consequence for treatment
Unusable as is. Two families of transformation are used: decomposition into calendar components (year, month, day of week, hour, weekend or holiday flag) and conversion to a relative duration with respect to a reference point (age in days, time since last event, time remaining until expiry). These constructions belong to feature engineering (chapter 024).
Point of caution
The presence of a temporal variable signals a possible chronological dependence. If the model is meant to predict the future from the past, the train / test split must be chronological, not random (chapter 027). A model trained on data that postdates the test data has been validated under conditions that cannot occur in production.
Rigorous definition
A variable whose values are natural-language character strings, of variable length and unconstrained structure.
Recognition test
Is the content written freely, with no finite set of levels? A customer comment is textual; a three-letter country code is categorical even though it is also stored as a string.
Examples
Review comment, listing description, email subject line, survey verbatim, free-text complaint reason.
Consequence for treatment
Vectorization is mandatory: occurrence counts, TF-IDF weighting, or a dense vector representation. Chapter 042 covers text classification by Bayesian methods; deeper natural language processing is outside the scope of this course.
Point of caution
A text column with low cardinality is in fact a mistyped categorical variable. The
distinguishing criterion is the number of distinct values relative to the number of
observations: df["column"].nunique().
The typology above is operational. It rests on a theoretical frame proposed by the psychophysicist S. S. Stevens in 1946, which classifies measurement scales by the mathematical operations they license.
Nominal scale — Values are labels and nothing more. Only equality is defined; no arithmetic operation is meaningful. Legitimate measure of central tendency: the mode.
Ordinal scale — Values are ordered, but the gaps are not interpretable. Equality and comparison are defined. Legitimate statistics: mode, median, quantiles.
Interval scale — Gaps are interpretable and constant, but the zero is conventional. Differences are meaningful, ratios are not. Canonical example: temperature in degrees Celsius, where 20 °C is not twice as hot as 10 °C. Legitimate statistics: mode, median, mean, standard deviation.
Ratio scale — The zero is absolute and means the absence of the quantity. Both differences and ratios are interpretable: EUR 40 really is twice EUR 20. All operations are legitimate, including the geometric mean and the coefficient of variation.
Scope of the frame
Stevens' classification supplies the rigorous criterion for which statistics are legitimate. The operational typology of section 4 maps onto it, with interval and ratio grouped together as numeric because the usual algorithms do not distinguish them.
Point of caution
The frame has been criticized, notably by Velleman and Wilkinson (1993), who dispute its mechanical application to forbid particular analyses. It nonetheless remains the pedagogical reference and the best available guardrail against meaningless computation.
| Scale | Relations defined | Zero | Legitimate statistics | Example |
|---|---|---|---|---|
| Nominal | Equality | Not applicable | Mode, counts, chi-square | City, brand |
| Ordinal | Equality, order | Not applicable | Mode, median, quantiles, rank correlation | Energy rating |
| Interval | Equality, order, difference | Conventional | Mean, standard deviation, linear correlation | Celsius temperature, calendar year |
| Ratio | Equality, order, difference, ratio | Absolute | All, including the geometric mean | Price, weight, duration |
A column stored as an integer is not necessarily a numeric variable. The deciding criterion is not the storage type but the nature of the quantity: do arithmetic operations have any business meaning?
| Column | Storage type | Actual nature | Disqualifying test |
|---|---|---|---|
| Postal code | Integer | Nominal categorical | The mean of two postal codes designates no place |
| Region code, state FIPS code | Integer | Nominal categorical | The number orders alphabetically, not geographically |
| Product code, store code | Integer | Nominal categorical | No business ordering relation between two codes |
| Quarter number (1 to 4) | Integer | Cyclic ordinal | Quarter 4 precedes quarter 1 of the following year |
| Rating (1 to 5) | Integer | Ordinal | The gap from 1 to 2 does not equal the gap from 4 to 5 |
| Numeric customer id | Integer | Identifier | No business meaning (section 3.4) |
| Industry classification code | String or integer | Hierarchical nominal | A multi-level nomenclature, to be exploited level by level |
| Year built | Integer | Interval numeric | The converse case: the difference is meaningful, the ratio is not |
What a misclassification costs. Treating a postal code as numeric leads a decision tree to produce splits such as "postal code below 44300", grouping together places with no geographic or socioeconomic proximity whatsoever: the model learns the arbitrary structure of the numbering scheme. A linear model, for its part, posits a monotone relationship between a place number and the target — an assumption with no foundation at all.
Point of caution. The correct transformation of a postal code is not necessarily a one-hot encoding of several thousand levels. The usual approaches are aggregation to a coarser level (county, metropolitan area), joining in territorial socioeconomic variables, or target encoding with the standard precautions against leakage (chapters 022 and 028).
| Type | Recognition test | Examples | Encoding required | Reference |
|---|---|---|---|---|
| Continuous numeric | Does an intermediate value make sense? | Price, salary, temperature, duration | None; scaling depending on the algorithm | Chapter 023 |
| Discrete numeric | Is it a count? | Number of rooms, number of calls | None; log transform if strongly skewed | Chapter 024 |
| Nominal categorical | Does ranking the levels make sense? No | City, brand, sector | One-hot; target encoding at high cardinality | Chapter 022 |
| Ordinal categorical | Does ranking make sense, with gaps not measurable? | Satisfaction, education, energy rating | Ordinal, with an explicit mapping | Chapter 022 |
| Binary | Exactly two levels? | Yes / no, active / inactive | 0 / 1, documented convention | Chapter 022 |
| Temporal | Does it designate an instant? | Date, timestamp | Calendar decomposition and relative durations | Chapters 024 and 027 |
| Textual | Is the content written freely? | Comment, description | Vectorization: counts, TF-IDF, embeddings | Chapter 042 |
Six operations are performed systematically, in this order. The order is not arbitrary: the dimensions govern how the types are read, the types govern how the statistics are read, and the target distribution governs the choice of metrics.
The outputs below come from a telecom churn dataset of 10,000 observations and 14 columns. Every block was executed; the printed text is the real output.
import pandas as pd
df = pd.read_csv("telecom_churn.csv")
print(df.shape)(10000, 14)Interpretation. 10,000 observations, 14 columns. The volume regime is nominal in the sense of section 1.5, with a favorable n / p ratio. That figure still has to be qualified by two later checks: the number of genuinely distinct observations (section 5.6) and the size of the minority class (section 5.5).
pd.set_option("display.max_columns", None)
print(df.head()) customer_id age region tenure_months plan_type contract \
0 C-00001 34 Metro 8 Mobile No_commitment
1 C-00002 67 South 45 Fiber 24_months
2 C-00003 29 East 2 Mobile No_commitment
3 C-00004 52 Metro 61 Fiber+TV 24_months
4 C-00005 41 West 17 Fiber 12_months
monthly_bill avg_data_gb support_calls satisfaction estimated_income \
0 29,90 8.4 0 4.0 1850.0
1 64,90 2.1 3 2.0 3120.0
2 19,90 14.7 1 NaN NaN
3 89,90 31.5 0 5.0 4470.0
4 49,90 6.8 7 1.0 2260.0
payment_method signup_date churned
0 Direct_debit 2024-06-12 0
1 Direct_debit 2021-09-03 0
2 Card 2024-12-20 1
3 Direct_debit 2020-02-28 0
4 Transfer 2024-01-15 1Interpretation. The output confirms the structure and already exposes two
formatting defects. monthly_bill uses a comma as the decimal separator, which
blocks numeric inference. signup_date will stay a character string until a
conversion is requested explicitly.
Point of caution. df.head() shows the first five rows of the file, which are
not a random sample. If the file is sorted by date or by region, those rows are not
representative of anything. df.sample(5, random_state=0) is a useful complement:
print(df.sample(5, random_state=0)) customer_id age region tenure_months plan_type contract \
9394 C-07639 50 Metro 7 DSL 24_months
898 C-05984 32 Metro 35 Fiber No_commitment
2398 C-03939 56 Metro 37 Fiber 12_months
5906 C-08708 27 West 68 Fiber No_commitment
2343 C-04730 57 Metro 33 Mobile 24_months
monthly_bill avg_data_gb support_calls satisfaction \
9394 29,90 NaN 1 NaN
898 49,90 11.1 0 4.0
2398 49,90 50.0 6 1.0
5906 69,90 6.2 3 NaN
2343 39,90 32.4 0 4.0
estimated_income payment_method signup_date churned
9394 2852.0 Direct_debit 2024-11-25 0
898 1782.0 Card 2022-07-22 1
2398 2520.0 Card 2022-06-05 1
5906 3164.0 Direct_debit 2019-11-18 0
2343 2115.0 Transfer 2022-09-28 0The random sample shows what the first five rows did not: missing values are common
in avg_data_gb and satisfaction, not exceptional.
df.info()<class 'pandas.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 14 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 customer_id 10000 non-null str
1 age 10000 non-null int64
2 region 10000 non-null str
3 tenure_months 10000 non-null int64
4 plan_type 10000 non-null str
5 contract 9964 non-null str
6 monthly_bill 10000 non-null str
7 avg_data_gb 8791 non-null float64
8 support_calls 10000 non-null int64
9 satisfaction 7412 non-null float64
10 estimated_income 9503 non-null float64
11 payment_method 10000 non-null str
12 signup_date 10000 non-null str
13 churned 10000 non-null int64
dtypes: float64(3), int64(4), str(7)
memory usage: 1.1 MBA note on the printed dtype. This output comes from pandas 3.x, which reports
string columns as str. Under pandas 2.x the same columns print as object. The
diagnosis below is identical in both cases: what matters is that seven columns are
held as text.
First reading — completeness. Any Non-Null Count below 10,000 signals missing
values: contract (36), avg_data_gb (1,209), satisfaction (2,588),
estimated_income (497). The magnitude determines the strategy: a column at 26%
missing is not handled like a column at 0.4% (chapter 018).
Second reading — types. A text dtype means the column was read as character data. Three cases must be separated.
| Column | Inferred type | Expected type | Diagnosis |
|---|---|---|---|
customer_id, region, plan_type, contract, payment_method | text | text | Correct: an identifier and four categorical variables |
monthly_bill | text | float64 | Badly inferred: comma decimal separator |
signup_date | text | datetime64 | Badly inferred: no conversion was requested |
Point of caution. A badly inferred type silently removes the column from
descriptive statistics and from numeric processing. monthly_bill is one of the
most explanatory variables of a churn phenomenon, and it will simply not appear in
df.describe() — as section 5.4 demonstrates. Worse, an automatic encoder would
treat it as a categorical variable with hundreds of levels. The usual causes are the
decimal separator, thousands separators, unit symbols, and the strings N/A, -,
or unknown.
df["monthly_bill"] = (
df["monthly_bill"].str.replace(",", ".", regex=False).astype(float)
)
df["signup_date"] = pd.to_datetime(df["signup_date"])Run before the type fix, describe covers only the columns pandas managed to read
as numbers:
print(df.describe(include="number").round(2)) age tenure_months avg_data_gb support_calls satisfaction \
count 10000.00 10000.00 8791.00 10000.00 7412.00
mean 46.31 32.16 12.92 1.70 3.40
std 15.39 24.45 9.76 2.08 1.13
min 18.00 1.00 0.00 0.00 1.00
25% 35.00 8.00 5.80 0.00 3.00
50% 46.00 29.00 10.50 1.00 3.00
75% 57.00 55.00 17.60 3.00 4.00
max 142.00 72.00 81.60 19.00 5.00
estimated_income churned
count 9503.00 10000.00
mean 2849.76 0.27
std 1463.46 0.44
min -5000.00 0.00
25% 1807.50 0.00
50% 2618.00 0.00
75% 3635.00 1.00
max 18400.00 1.00monthly_bill is absent. Nothing warned about it. That is exactly the failure mode
described in section 5.3, and it is why the type fix belongs before this step, not
after. Once the two columns are converted, the same call returns the full table:
print(df.describe(include="number").round(2)) age tenure_months monthly_bill avg_data_gb support_calls \
count 10000.00 10000.00 10000.00 8791.00 10000.00
mean 46.31 32.16 51.31 12.92 1.70
std 15.39 24.45 24.31 9.76 2.08
min 18.00 1.00 19.90 0.00 0.00
25% 35.00 8.00 29.90 5.80 0.00
50% 46.00 29.00 49.90 10.50 1.00
75% 57.00 55.00 69.90 17.60 3.00
max 142.00 72.00 114.90 81.60 19.00
satisfaction estimated_income churned
count 7412.00 9503.00 10000.00
mean 3.40 2849.76 0.27
std 1.13 1463.46 0.44
min 1.00 -5000.00 0.00
25% 3.00 1807.50 0.00
50% 3.00 2618.00 0.00
75% 4.00 3635.00 1.00
max 5.00 18400.00 1.00What this operation is for: detecting impossible values, that is, values outside the business domain of definition of the variable. These are not extreme values, they are errors.
| Finding | Value | Diagnosis |
|---|---|---|
age maximum | 142 | Out of domain: data entry error, sentinel value, or a mis-recorded date of birth |
estimated_income minimum | -5,000 | Out of domain: sign error, or a repurposed sentinel code |
estimated_income maximum | 18,400 | Suspiciously round and far above the 75th percentile of 3,635: a cap or a sentinel, to be checked against the source system |
avg_data_gb maximum | 81.6 | Extreme but plausible: a legitimate outlier, to be examined (chapter 021) |
satisfaction count | 7,412 | Partial completeness: 26% missing |
churned mean | 0.27 | The mean of a binary variable is the proportion of the positive class |
Point of caution on sentinel values. The values -1, 999, 9999, -9999, a
0 on a strictly positive variable, and the date 1900-01-01 are conventional codes
signaling the absence of information in many legacy systems. Treated as real values,
they distort means and create artificial relationships.
Point of caution on scope. df.describe() covers numeric columns by default.
Examining the categorical columns requires df.describe(include="object"), which
returns count, cardinality, and most frequent level:
print(df.describe(include="object")) customer_id region plan_type contract payment_method
count 10000 10000 10000 9964 10000
unique 9857 5 4 3 3
top C-06883 Metro Mobile 24_months Direct_debit
freq 2 3103 3363 3649 5787This single table settles three questions at once. region, plan_type,
contract, and payment_method have cardinalities of 5, 4, 3, and 3 — all
one-hot encodable without difficulty. And customer_id has 9,857 unique values
over 10,000 rows, with a most frequent value appearing twice. An abnormally high
cardinality in this table reveals an undeclared identifier or a mistyped text
column; here it reveals something more serious, taken up in section 5.6.
This is the check with the best ratio of information to cost. It takes one line of code and determines the problem type, the choice of metrics, the splitting strategy, and the baseline model.
print(df["churned"].value_counts(normalize=True).round(4))
print(df["churned"].value_counts())churned
0 0.7347
1 0.2653
Name: proportion, dtype: float64
churned
0 7347
1 2653
Name: count, dtype: int64Nature of the problem. Two levels: binary classification. A continuous target would mean regression, a target with more than two levels multiclass classification (chapter 006).
Level of imbalance. A 73 / 27 ratio is a moderate imbalance, manageable with a stratified split and class weighting.
Baseline model. A classifier that always predicts the majority class reaches 73.47% accuracy without learning anything. Every subsequent performance figure is compared against that threshold, not against 50%. A model reporting 75% delivers a marginal gain of 1.5 points.
Usable count. The positive class holds 2,653 observations. That number, not 10,000, bounds the model's ability to characterize churn.
| Majority / minority ratio | Qualification | Methodological consequence |
|---|---|---|
| 50 / 50 to 65 / 35 | Balanced | No specific treatment; accuracy remains interpretable |
| 65 / 35 to 90 / 10 | Moderate imbalance | Stratified split, class weighting, per-class metrics |
| 90 / 10 to 99 / 1 | Strong imbalance | Accuracy disqualified; use PR-AUC, recall, precision; tune the threshold |
| Beyond 99 / 1 | Extreme imbalance | Controlled resampling, cost-sensitive learning, anomaly detection |
Imbalance is developed in chapter 050, the associated choice of metrics in chapters 053 and 064, threshold tuning in chapter 062.
Point of caution. value_counts() drops missing values by default. Calling
df["churned"].value_counts(dropna=False) is essential: a partially unpopulated
target signals a defect in how the dataset was assembled, since those observations
can serve neither for training nor for evaluation. Here the check comes back clean:
print(df["churned"].value_counts(dropna=False))churned
0 7347
1 2653
Name: count, dtype: int64missing = df.isnull().sum()
missing = missing[missing > 0].sort_values(ascending=False)
print(missing)
print((missing / len(df) * 100).round(2))satisfaction 2588
avg_data_gb 1209
estimated_income 497
contract 36
dtype: int64
satisfaction 25.88
avg_data_gb 12.09
estimated_income 4.97
contract 0.36
dtype: float64Interpretation. The rate matters more than the raw count. A common rule of thumb places around 5% the level below which simple imputation is generally harmless, and around 40 to 50% the level above which the column's usefulness should be questioned outright. These are landmarks, not rules.
Point of caution. Absence of a value frequently carries information. An
unpopulated satisfaction marks a customer who has never answered a survey, which
is itself a behavioral signal. Creating a binary indicator satisfaction_missing
before imputation preserves that information (chapter 018).
print("Strictly duplicated rows:", df.duplicated().sum())
print("Duplicated identifiers :", df["customer_id"].duplicated().sum())
print("Distinct customers :", df["customer_id"].nunique())Strictly duplicated rows: 0
Duplicated identifiers : 143
Distinct customers : 9857Interpretation — and this is the most consequential result of the whole inspection. No row is duplicated in full, which rules out a concatenation accident. But 143 identifiers appear more than once: the dataset holds 9,857 distinct customers described across 10,000 rows.
The unit of analysis is therefore not exactly the customer. Three questions follow. Do these rows correspond to several observation dates for the same customer, to distinct contracts held by one customer, or to a consolidation defect? Depending on the answer, the split must be performed by group (chapter 027), and section 2.3 applies in full.
Note also that df.duplicated().sum() alone would have returned 0 and closed the
question. It is the check on the business key that opened it.
Strict duplicate
Two rows identical across every column. Detection: df.duplicated(). Usual origin:
repeated concatenation, a join with the wrong cardinality, a partial reload.
Functional duplicate
Two rows designating the same real-world entity without being identical, because of
a difference in case, spacing, format, or timestamp. Detection:
df.duplicated(subset=[...]) on the columns forming the business key.
Consequence for modeling
A strict duplicate present on both sides of a random split guarantees a correct prediction on the corresponding test observation, by memorization. The measured performance is mechanically overstated. This is a form of leakage (chapter 028).
Point of caution
Removing duplicates is not automatically justified. Two transactions of the same amount, at the same merchant, on the same day, can both be genuine. The decision requires the business key, never the mere identity of the values.
FIRST-CONTACT CHECKLIST FOR A DATASET
1. What does one row represent, exactly? State the answer in a full
sentence, validated by the business.
2. How many observations (n) and how many columns (p)? Is the n / p
ratio compatible with the ambition of the model?
3. Does one entity appear on several rows? Check nunique() on the
business key, not only duplicated().
4. Which column is the target, and is it fully populated?
5. What is the distribution of the target? Derive the problem type,
the level of imbalance, and the baseline model.
6. What is the functional role of each column: identifier, feature,
target, or leakage variable?
7. What is the type of each variable, and does the type inferred at
read time match the real nature of the quantity?
8. Which columns have missing values, at what rate, and is the
missingness itself informative?
9. Are the extreme values plausible? Separate the impossible value
from the legitimate outlier.
10. Are sentinel values used to signal absence of information
(-1, 999, 9999, 1900-01-01)?
11. Is there a time component, and does it force a chronological
split rather than a random one?
12. Will every feature be available and populated AT THE MOMENT the
prediction has to be produced?The twelfth point is the one most often skipped. The first eleven concern the file as it presents itself and can be verified in code. The twelfth concerns the business process that produces the data, and it can only be verified by talking to the people who operate the system.
A variable can be perfectly populated in the historical extract and unavailable at prediction time: populated later in the process, computed by an overnight batch, or entered by an operator in reaction to the very event the model is supposed to anticipate. A model built on such variables shows excellent validation performance and is unusable in production (chapter 028).
Each of the three cases below presents a realistic extract, then the structured analysis to produce before any modeling.
| customer_id | age | tenure_months | plan_type | contract | monthly_bill | support_calls | satisfaction | signup_date | churned |
|---|---|---|---|---|---|---|---|---|---|
| C-00001 | 34 | 8 | Mobile | No_commitment | 29,90 | 0 | 4 | 2024-06-12 | 0 |
| C-00002 | 67 | 45 | Fiber | 24_months | 64,90 | 3 | 2 | 2021-09-03 | 0 |
| C-00003 | 29 | 2 | Mobile | No_commitment | 19,90 | 1 | — | 2024-12-20 | 1 |
| C-00004 | 52 | 61 | Fiber+TV | 24_months | 89,90 | 0 | 5 | 2020-02-28 | 0 |
| C-00005 | 41 | 17 | Fiber | 12_months | 49,90 | 7 | 1 | 2024-01-15 | 1 |
| Element of the analysis | Determination |
|---|---|
| What one observation represents | One customer, described as of the extraction date, with churn status observed over the following twelve months |
| Identifiers to exclude | customer_id: removed from the features, kept for traceability and for grouping |
| Target | churned, binary |
| Problem type | Binary classification, imbalanced target at roughly 27% positives |
| Features | age, region, tenure_months, plan_type, contract, monthly_bill, avg_data_gb, support_calls, satisfaction, estimated_income, payment_method, plus variables derived from signup_date. The extract above shows only a subset of the file's 14 columns |
| Variable types | Continuous: monthly_bill. Discrete: tenure_months, support_calls, age. Nominal: plan_type, region, payment_method. Ordinal: contract, satisfaction. Temporal: signup_date |
| Points of caution | monthly_bill is mistyped (comma separator). satisfaction is 26% missing, and the missingness is informative. signup_date is redundant with tenure_months and should be decomposed. 143 identifiers repeat, so the unit of analysis is not cleanly the customer. The churn_date column of the source file is a leakage variable, to be excluded without exception |
On the ordinality of contract. The levels No_commitment, 12_months, and
24_months are ordered by increasing commitment length, so an ordinal encoding is
justified. Point of caution: an automatic encoder based on alphabetical order
would produce 12_months < 24_months < No_commitment, placing the absence of any
commitment at the highest level — which reverses the business meaning of the
variable entirely. The mapping must be stated explicitly.
| listing_id | city | postal_code | area_sqm | n_rooms | floor | year_built | energy_class | has_elevator | sale_date | sale_price |
|---|---|---|---|---|---|---|---|---|---|---|
| A-10471 | Lyon 3 | 69003 | 68 | 3 | 4 | 1972 | D | 1 | 2024-03-14 | 331,000 |
| A-10472 | Villeurbanne | 69100 | 42 | 2 | 1 | 2015 | B | 1 | 2024-03-18 | 218,500 |
| A-10473 | Lyon 6 | 69006 | 105 | 5 | 2 | 1930 | E | 0 | 2024-04-02 | 712,000 |
| A-10474 | Bron | 69500 | 77 | 4 | 0 | 1988 | D | 0 | 2024-04-11 | 249,000 |
| A-10475 | Lyon 3 | 69003 | 31 | 1 | 6 | 2019 | A | 1 | 2024-05-06 | 189,900 |
| Element of the analysis | Determination |
|---|---|
| What one observation represents | One completed property transaction, not one property: a property sold twice legitimately occupies two rows |
| Identifiers to exclude | listing_id |
| Target | sale_price, continuous and strictly positive |
| Problem type | Regression |
| Features | city, area_sqm, n_rooms, floor, year_built, energy_class, has_elevator, plus variables derived from sale_date |
| Variable types | Continuous: area_sqm. Discrete: n_rooms, floor. Interval: year_built. Nominal: city. Ordinal: energy_class (A to G). Binary: has_elevator. Temporal: sale_date |
| Points of caution | postal_code is a numeric false friend: nominal categorical, redundant with city, to be excluded or aggregated. year_built is an interval variable; converting it to the property's age at the sale date is preferable. energy_class is ordinal, with performance decreasing from A to G — an order that must be stated. The target is strongly right-skewed, which justifies examining a model on its logarithm |
On temporal dependence. Property markets drift in price level. A model trained on 2024 transactions and applied in 2026 is exposed to concept drift (chapter 082). If the intended use is to value future transactions, the split must be chronological: train on the older periods, evaluate on the most recent one (chapter 027).
On the target's skew. Prices in the extract range from EUR 189,900 to EUR
712,000, and the underlying distribution has a long right tail. Fitting on
log(sale_price) converts multiplicative error into additive error, which usually
matches the business reading — a valuation is judged as a percentage error, not as
an absolute one. The consequence for the metric is covered in chapter 074.
| transaction_id | card_id | timestamp | amount | merchant | mcc_code | country | channel | seconds_since_previous | is_fraud |
|---|---|---|---|---|---|---|---|---|---|
| T-9910001 | K-4471 | 2025-02-11 08:14:22 | 12.40 | Vidal Bakery | 5462 | FR | Contactless | 43,120 | 0 |
| T-9910002 | K-4471 | 2025-02-11 08:19:07 | 1,890.00 | ElectroDirect | 5732 | LT | Online | 285 | 1 |
| T-9910003 | K-4471 | 2025-02-11 08:21:55 | 1,940.00 | ElectroDirect | 5732 | LT | Online | 168 | 1 |
| T-9910004 | K-2038 | 2025-02-11 09:02:11 | 67.30 | Aral Station | 5541 | DE | Chip | 61,400 | 0 |
| T-9910005 | K-7712 | 2025-02-11 09:03:48 | 8.90 | Metro Line 4 | 4111 | FR | Contactless | 22,010 | 0 |
| Element of the analysis | Determination |
|---|---|
| What one observation represents | One card transaction at one dated instant. The unit of analysis is the transaction, neither the card nor the cardholder |
| Identifiers to exclude | transaction_id and card_id from the features. card_id must be kept as the grouping key |
| Target | is_fraud, binary, with a positive rate on the order of 0.1% to 0.5% in real populations |
| Problem type | Binary classification with extreme imbalance |
| Features | amount, mcc_code, country, channel, seconds_since_previous, plus variables derived from timestamp and rolling aggregates per card |
| Variable types | Continuous: amount, seconds_since_previous. Nominal: mcc_code, country, channel, merchant. Temporal: timestamp |
| Points of caution | Three pitfalls compound, developed below. merchant has a cardinality of several hundred thousand levels and cannot be one-hot encoded. mcc_code is a numeric false friend: a categorical nomenclature |
First pitfall — extreme imbalance. At 0.17% fraud, a classifier that always predicts "not fraud" reaches 99.83% accuracy. The metric is disqualified from the outset. Evaluation rests on precision, recall, and the area under the precision-recall curve, the ROC curve being too optimistic under extreme imbalance. The decision threshold becomes a control parameter, set by the relative cost of a missed fraud and an unjustified block.
Second pitfall — the repeated entity. One card produces many transactions. A
random split puts transactions from the same card on both sides of the partition:
the model learns the spending habits of cards it will meet again at evaluation time,
whereas the intended use is to rule on cards whose recent history was never
learned. Grouping by card_id is mandatory.
Third pitfall — temporal order. Fraud patterns move quickly, in waves tied to active campaigns. A random split trains the model on transactions that postdate the evaluation ones: the model knows patterns that, in production, would not yet exist. The split must be strictly chronological, with the evaluation window later than the training window.
Point of caution on the compounding. All three constraints apply at once. The split must be chronological and respect groups, which rules out the standard splitters and forces a specific procedure: cut on time first, then drop the cards that appear on both sides of the boundary. This is a case where analyzing the dataset determines the entire validation architecture, before a single model is fitted.
An examiner wants to assess a candidate's ability to diagnose a rare disease.
If the exam is built so that one case in six hundred is pathological, a candidate who answers "healthy" every time scores 99.8% with no medical competence at all. That is the first pitfall.
If the exam presents patient files the candidate already worked through during training, the examiner is measuring memory, not diagnosis. That is the second pitfall.
If, during training, the examiner hands the candidate the cases that will appear in a later session, the examiner is measuring a capability that will never exist in practice. That is the third pitfall.
The three biases add up. An assessment that combines all three produces a high score with no predictive value whatsoever.
A dataset of 50,000 rows is presented as containing 50,000 customers. It actually contains 50,000 contracts spread over 12,000 customers. The effective volume is divided by four, the random split is invalid, and the business scope of the predictions is not the one that was announced.
Correct formulation : "Before any manipulation, the question to settle is what a row represents exactly, and whether one entity can appear on several rows. The answer determines the effective volume, the splitting strategy, and how the predictions are to be read."
An identifier is arbitrary by construction. It has no explanatory power, it absorbs the spurious correlations induced by the order of assignment, and its near-uniqueness lets high-capacity models memorize individual observations.
Correct formulation : "Identifiers are excluded from the features while being kept in the dataset, for traceability and for grouping at split time. When an identifier encodes business information, that information is extracted into a dedicated variable and the raw identifier is then dropped."
This check costs one line of code and determines the problem type, the level of imbalance, the baseline model, the choice of metrics, and the splitting strategy. Skipping it leads to reading 97% accuracy as a performance, when it may be lower than that of a constant classifier.
Correct formulation : "The distribution of the target is examined before any training. It fixes the baseline against which every later performance figure must be compared."
A postal code, a region code, a product code, or a classification code is stored as an integer and is categorical by nature. Treated as numeric, they lead a tree to produce meaningless splits ("postal code below 44300") and a linear model to posit a monotone relationship between a place number and the target.
Correct formulation : "Storage type does not determine the nature of a variable. The criterion is the business meaning of arithmetic operations: if the mean of two values designates nothing, the variable is categorical."
A file with fourteen columns does not give you fourteen features. It gives you fourteen columns whose functional role remains to be determined: identifier, feature, target, or leakage variable. Leakage columns produce near-perfect apparent performance in validation and a worthless model in production.
Correct formulation : "The role of every column is determined individually. The admission criterion for the feature set is the actual availability of the value at the instant the prediction must be produced."
A strict duplicate present on both sides of a split guarantees a correct prediction
by memorization and overstates measured performance. A functional duplicate — same
entity, non-identical rows — is not detected by df.duplicated() and requires a
check on the business key.
Correct formulation : "The check operates at two levels: strictly identical rows, and rows designating the same real-world entity. The second level requires knowledge of the business key and cannot be automated without it."
A million observations with a mislabeled target, with features unavailable in production, or in which a single entity accounts for 40% of the rows, does not produce a better model than ten thousand clean, correctly framed observations.
Correct formulation : "Volume governs the variance of the estimate. It corrects no bias. A framing defect reproduces itself identically at every scale."
STRUCTURE
A dataset is an n × p matrix.
n = number of observations (the rows)
p = number of variables (the columns)
X = feature matrix, y = target vector
TERMINOLOGY OF THE ROW
observation · individual · instance · sample (ML sense) ·
record · row · example · data point
Only one term is ambiguous: sample, which also denotes
a subset of a population in the statistical sense.
TERMINOLOGY OF THE COLUMN
variable · attribute · field · column · dimension ·
feature · predictor · covariate
variable and column describe; feature assigns a role.
THE FOUR FUNCTIONAL ROLES OF A COLUMN
identifier -> exclude, keep as a grouping key
feature -> submit to the model
target -> isolate in y
leakage variable -> exclude as an absolute priority
TYPOLOGY OF VARIABLES
continuous numeric · discrete numeric
nominal categorical · ordinal categorical · binary
temporal · textual
Stevens' scales: nominal · ordinal · interval · ratio
FALSE FRIENDS
postal code, region code, product code, classification code:
numeric in appearance, categorical in nature.
INITIAL INSPECTION PROTOCOL
df.shape · df.head() · df.info() · df.describe()
df[target].value_counts(normalize=True)
df.isnull().sum() · df.duplicated().sum()
df[key].nunique()
THE FIRST QUESTION
What does one row represent, exactly,
and can one entity appear on several rows?
THE HIGHEST-YIELD CHECK
The distribution of the target.
THE MOST FREQUENTLY OMITTED POINT
Will every feature be available at the moment
the prediction has to be produced?Summary statement
A dataset is a matrix whose rows are observations and whose columns are variables. Reading it professionally means establishing what a row represents, assigning every column a functional role rather than assuming it is explanatory, placing every variable in a typology that dictates its treatment, and verifying that every retained variable will actually be available at the instant of prediction.
Associated quizzes
005.1-quiz-dataset.md005.2-quiz-observation-unit-of-analysis.md005.3-quiz-variable-roles.md005.4-quiz-variable-typology.md005.5-quiz-initial-inspection.md005.6-quiz-case-studies.mdNext chapter : 006.0-features-and-target.md