Web Scraping Meta Tags Without Getting Blocked — Lessons Learned

I've spent the last few months building a system that extracts meta tags from URLs at scale. Along the way I hit every wall you can imagine — rate limits, CAPTCHAs, bot detection, encoding nightmares, and HTML so malformed it would make a parser cry. Here's everything I learned, so you don't have to learn it the hard way. The Simple Version (That Breaks Immediately) Extracting meta tags seems trivial: const res = await fetch ( url ); const html = await res . text (); const title = html . match ( /<title> ( .* ?) < \/ title>/ )?.[ 1 ]; This works for about 60% of websites. The other 40% will teach you humility. Problem 1: Bot Detection Many sites block requests that don't look like a real browser. What Gets You Blocked Missing or generic User-Agent header No Accept , Accept-Language , or Accept-Encoding headers Requesting from cloud provider IP ranges (AWS, GCP, Azure) Making too many requests too fast Missing TLS fingerprint characteristics What Works Set headers that look like a real