How We Beat Cloudflare to Scrape Indeed.com — From 403 Blocked to 16 Jobs in Production

A real-world story of building a job scraper that works on datacenter servers, not just localhost. The Task Our CTO dropped a message in Slack: "New scraper tool for Indeed.com — this will have high volume of jobs. Daily summary at 7am. Keywords: software engineer, php developer, devops. Employer-only filter." Simple enough, right? Indeed.com has thousands of jobs. We just need to scrape them, store them in our database, and send a Slack summary every morning. What followed was a 3-day rabbit hole through Cloudflare challenges, Tor exit nodes, anti-detection browsers, CAPTCHA solvers, and residential proxies — before we finally landed on a clean, cost-effective solution. Attempt 1: The Mobile API (JobSpy) Our first instinct was to avoid the browser entirely. JobSpy is a Python library that hits Indeed's mobile app API — the same backend the iPhone/Android app uses. No browser, no Cloudflare. Our Server → Indeed Mobile API → ✅ Structured job data The problem: The mobile API doesn't supp