
Website changed a div class. Scraper returned nothing.
Spent 3 weeks building a scraper that pulled job postings from a niche job board. Worked perfectly. Ran it daily, collected data, built a dashboard on top of it. Then one Tuesday morning the script finished in 4 seconds instead of 2 minutes. Zero results. Website redesigned their listings page. Changed .job-card to .listing-item . Beautiful Soup kept looking for the old class. Found nothing. Script logged success and moved on like an idiot. No error. No warning. Just silence. Thought I could just update the selector: # Before jobs = soup . find_all ( ' div ' , class_ = ' job-card ' ) # After jobs = soup . find_all ( ' div ' , class_ = ' listing-item ' ) Worked for 2 days. They changed it AGAIN to .post-container . I was not doing this every week. Ended up making it more flexible Instead of hardcoding class names I targeted the parent container that stayed consistent: from bs4 import BeautifulSoup import requests response = requests . get ( ' https://example-job-board.com/listings ' ) s
Continue reading on Dev.to Tutorial
Opens in a new tab



