The Web Scraping Checklist I Wish I Had When I Started (21 Steps)

After building 77 scrapers for production use, I realized I follow the same 21 steps every time. This is the checklist I give to every developer on my team. Before You Write Any Code [ ] 1. Check for an official API. 60% of 'scraping' projects don't need scraping at all. Check the site's /api/ , developer docs, or look for application/json responses in DevTools. [ ] 2. Check robots.txt. Visit example.com/robots.txt . If your target path is Disallow , proceed with caution. [ ] 3. Read the Terms of Service. Search for "scraping", "automated", "bot". Some sites explicitly prohibit it. [ ] 4. Check if the data is available elsewhere. Common Crawl, Wayback Machine, public datasets (data.gov, Kaggle) might already have what you need. [ ] 5. Decide: HTTP client or browser? If the page works with JavaScript disabled → use httpx / requests . If not → use Playwright. Writing the Scraper [ ] 6. Start with one page. Get it working perfectly for one URL before scaling. [ ] 7. Use CSS selectors, not