Hyperparameter Optimization: Grid vs Random vs Bayesian

You've trained a Random Forest and it works — 85% accuracy out of the box. But you used the default hyperparameters. What if n_estimators=500 with max_features=0.3 and min_samples_leaf=10 pushes that to 91%? Only one way to find out: search. The problem is combinatorial. Our Random Forest has 4 hyperparameters. If you try 10 values for each in a grid, that's $10^4 = 10{,}000$ combinations. Each combination requires 5-fold cross-validation. That's 50,000 model fits — and we only have 4 dimensions. Neural networks routinely have 10–20 tunable hyperparameters, where exhaustive search is physically impossible. This post compares three strategies of increasing sophistication: Grid Search — try every combination on a predefined grid Random Search — sample combinations at random (surprisingly effective) Bayesian Optimization — build a model of the objective and use it to choose the next point intelligently We'll run all three on the same classification task, using the same Random Forest and t