An Introduction to the Architectures Powering the Current LLMs

Large Language Models (LLMs) have rapidly taken the spotlight in a wide range of fields over the past few years. At Pruna, the focus has been clear: make these models smaller, faster, cheaper, and greener. To make this possible, the team has explored and provided different optimization techniques, from caching and model compilation to advanced quantization and beyond. | For an overview of AI model optimization techniques, see this blog . However, these individual optimizations are just pieces of a much larger machine. We must lift the hood and examine the engine to understand how it works. This blog post will provide an overview, not attempting to cover every mathematical detail, but focusing on the main intuition, of the key architectures powering today’s language models: Autoregressive Models, State-Space Models, Diffusion-based Models, and Liquid Neural Networks. Where It All Begins: Tokenizers and Embeddings Before we dive into the intricate inner workings, it’s worth remembering t