
How long should we run A/B test? (or "how to define MDE")
Most teams don't struggle with the math — they struggle with the question before the math. You open a calculator, plug in some numbers, get a sample size. But where did those numbers come from? Usually: a gut feeling, a copied benchmark, or whatever the LLM suggested. The thing is, there's a clear framework for this. And once you see it, the calculator stops being a black box. The framework: MDE is a product decision, not a statistical parameter Everything in sample size calculation flows from one number: the Minimum Detectable Effect — the smallest improvement you want to be able to detect. Most people treat MDE as a technical input. It's not. It's the answer to a product question: "What's the smallest lift that would justify shipping this variant?" If your team spent two sprints building a feature, it's probably not worth shipping unless it moves conversion by at least 5%. That's your MDE. Not 1%, not 0.5% — because even if those lifts are real, they don't change your decision. This
Continue reading on Dev.to
Opens in a new tab



