dReLU Sparsification: Recovering LLM Performance with 150B Token Pretraining

Learn how dReLU-based ReLUfication restores model capabilities for Mistral-7B and Mixtral-47B. Discover the high-quality pretraining datasets and mixture ratios used to achieve elite activation sparsity.