Tokens

🧠 Tokens in Transformers — Developer Notes 🔹 What is a Token? A token is the smallest unit of text that a transformer model processes. It is created by a tokenizer and then converted into numerical IDs before entering the model. ⚠️ Important: Token ≠ always a full word. 🔹 What Can Be a Token? Depending on the tokenizer, a token may be: whole word subword (most common) character punctuation special symbol ✅ Modern transformers mainly use subword tokenization . 🔹 Example Sentence: I like eating apples Possible subword tokens: [I] [like] [eat] [##ing] [apple] [##s] 🔹 Transformer Processing Pipeline Raw Text → Tokenizer → Tokens → Token IDs → Embeddings → Transformer Neural networks only understand numbers, so tokens must be converted to IDs and then to vectors. 🔹 Why Tokenization Is Needed Tokenization helps to: reduce vocabulary size handle unknown words capture morphology improve generalization enable efficient training 🔹 Special Tokens (Encoder Models) Typical encoder input: [CLS] I li