What the Hell is a Token?

Months after ChatGPT launched, I still could not have told you what a token was. I had been using it since the first public launch and was basically having novel-long conversations with it. I had no idea that every time I hit "enter," my text was being chopped into pieces before the model even looked at it. It turns out, those pieces (tokens) determine your usage limits, how much the AI can remember, and why it sometimes seems to forget things you told it. So. Tokens. Here is what I wish I understood earlier. They are not words I assumed "one token = one word," but that is not actually the case. A token is a chunk of text; it may be a whole word, part of a word, or punctuation. The word "hamburger" gets split into two tokens: h and amburger . Not "ham" and "burger". The splits are not based on syllables, like you might expect. Here are a few more to make the point: "infrastructure" becomes inf and rastructure . "Unbelievable" becomes three tokens: un , belie , and vable . These splits