The Ghost in the Tokenizer: How Subword Tokenization Invisibly Shapes What Your Prompt 'Means' to the Model

You type "unexpectedly beautiful." The AI understands. But does it? Between your keystroke and its understanding lies a hidden layer, a ghost in the machine that decides how to slice your words into digestible pieces. "Unexpectedly" might become ["un", "expect", "edly"]. "Beautiful" might become ["beaut", "iful"]. And in that slicing, meaning shifts. Associations form. The ghost has touched your prompt. This ghost is the tokenizer, and it's one of the most overlooked yet powerful factors in prompt engineering. The tokenizer doesn't care about your words; it cares about your tokens - the subword units that your prompt gets broken into before the model ever sees it. And savvy prompters are learning to speak not just to the model, but to the tokenizer itself. Let's pull back the curtain on this invisible layer. By the end, you'll understand how tokenization shapes meaning, why some prompts fail at the character level, and how to exploit this knowledge for finer control over your outputs.