From Bag of Words to Lexicons: A Simple Journey Through NLP Basics

Let me walk you through how machines learned to understand text. Nothing fancy. Just concepts. Bag of Words: The First Baby Step Bag of Words was exactly what it sounds like. You take a sentence, throw all words into a bag, shake it up, and count how many times each word appears. "I love chai" and "chai love I" look identical to Bag of Words. Same words. Same counts. Order doesn't matter. The idea was simple. If a word appears more times, it probably matters more. If positive words show up frequently, the text might be positive. The problem? Context disappears. "Not good" and "good" use the same words. But they mean opposite things. Bag of Words couldn't tell the difference. It just counted. Stop Words: Cleaning Up the Noise Early on, people noticed something obvious. Words like "the", "is", "at", "which" were showing up everywhere but adding zero value. Stop words are the filler words you remove before doing any actual work. The, a, an, and, but, or, for, so, to, from, with, by, at, i