Week 8 — Lesson 2: Bag-of-words and TF-IDF

3 min

NorthPeak technicians describe each incident in a short note. To predict the incident category from these notes, a model needs numbers, not sentences. The simplest recipe counts words: one column per word, one row per note. TF-IDF then lowers the weight of words such as the that appear in every note. This lesson builds that table on 108 training descriptions from incidents.csv and shows its limits.

Word counts and TF-IDF

The simplest way to turn text into numbers is to count words. Build a vocabulary: one column per distinct token. Then one row per text, with the count of each token. This table is a bag-of-words. The word order is lost, like words shaken in a bag.

Preview — the rest of the lesson is for enrolled readers.

Already enrolled with a code?

Your access is tied to your account, not to this link. Sign in with the same email you used in class: your course is waiting, no need to enter the code again.

Sign inNo account yet? Create one
This lesson is part of the “Week 8 — NLP and text representations” module

The first modules of the course are open to everyone. For the rest you have three options: buy this course once and for all, subscribe, or enter the code handed out in class.

Are you a student on this course?

The code is tied to your account: sign in or create an account and it will be applied automatically when you come back.

No account yet? Create one