Week 8 — Lesson 1: Tokenization and normalization

2 min

NorthPeak technicians write free text. The first incident of data/clean/incidents.csv reads Overheating alarm on M001. Cooling fan running but airflow blocked by dust. A model computes with numbers, so this sentence must first be cut into small pieces and cleaned. Every cleaning step makes the pieces more uniform, and every step also loses something. This lesson covers tokenization, normalization, and what to check before you trust the result.

Preview — the rest of the lesson is for enrolled readers.

Already enrolled with a code?

Your access is tied to your account, not to this link. Sign in with the same email you used in class: your course is waiting, no need to enter the code again.

Sign inNo account yet? Create one
This lesson is part of the “Week 8 — NLP and text representations” module

The first modules of the course are open to everyone. For the rest you have three options: buy this course once and for all, subscribe, or enter the code handed out in class.

Are you a student on this course?

The code is tied to your account: sign in or create an account and it will be applied automatically when you come back.

No account yet? Create one