Web Scraping Anti-Bot Guide: Delays, User-Agents, and When to Use Proxies

Getting blocked is the #1 frustration in web scraping. Here's how to avoid it. Rule 1: Always Set User-Agent const headers = { ' User-Agent ' : ' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' }; Without a User-Agent, many sites return 403 immediately. Rule 2: Add Random Delays const delay = ( ms ) => new Promise ( r => setTimeout ( r , ms )); for ( const url of urls ) { const data = await scrape ( url ); await delay ( 1000 + Math . random () * 3000 ); // 1-4s random } Fixed delays get detected. Random delays look human. Rule 3: Rotate User-Agents const agents = [ ' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' , ' Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ' , ' Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' ]; const randomAgent = agents [ Math . floor ( Math . random () * agents . length )]; Rule 4: Handle Rate Limits Gracefully async function fetchWithRetry ( url , maxRetries = 3 ) { for ( let i = 0 ; i < maxRetries ; i