Python Rate Limiting for Web Scrapers: Best Practices 2026

Rate limiting is one of the most critical aspects of web scraping. Without proper rate limiting, your scraper will get blocked, your IP banned, and you'll waste time debugging instead of collecting data. Here's a comprehensive guide to implementing rate limiting in your Python scrapers. Why Rate Limiting Matters Avoid IP bans : Websites detect and block aggressive scrapers Respect servers : Don't crash someone's website Better data : Rate-limited scrapers collect MORE data long-term Ethical scraping : Be a good citizen of the internet Token Bucket Rate Limiter A sophisticated approach that allows burst requests while maintaining an average rate: import time import threading import random class TokenBucket : def __init__ ( self , rate , capacity ): self . rate = rate self . capacity = capacity self . tokens = capacity self . last_refill = time . monotonic () self . lock = threading . Lock () def acquire ( self ): with self . lock : now = time . monotonic () elapsed = now - self . last_r