Kafka partitioning: the part that decides whether your design works

Once you know the basics of Kafka, partitioning is where the real design work starts. Topics, consumers, offsets, and producer APIs are straightforward enough. Partitioning is where correctness, throughput, and operational behavior meet. It is also where teams make decisions that look harmless early on and become expensive later. The first thing to keep in mind is that partitioning is not just a scaling mechanism. It is also Kafka’s ordering model. Kafka only guarantees order within a single partition. That means every decision about how records are assigned to partitions is also a decision about which records are ordered relative to each other, and which are not. How records get routed to partitions When a producer writes a record to Kafka, the partition is chosen in one of a few ways. The common case is keyed routing. If the record has a key, the producer hashes that key and uses the result to choose a partition. The exact hashing details depend on the client and partitioner implemen