Backfilling Is Harder Than Scraping: Lessons From Rebuilding 6 Months of Missing Data

Most scraping systems are designed for the present. fetch parse store Repeat. But production systems don’t fail in real time. They fail silently — and you only notice weeks later. The problem: missing history We ran into this after a pipeline issue. A scraper had been “working” for months, but due to a logic bug, it skipped: ~40% of updates over a 6-month period No crashes. No alerts. Just… gaps. And suddenly we had a new problem: How do you reconstruct data that was never collected? Why backfilling is fundamentally different Scraping live data is easy (relatively). Backfilling is not. Because the web is not static. When you go back in time, you’re dealing with: overwritten content expired listings mutated pages cached or partial states You’re not fetching history. You’re trying to infer it. The naive approach (that failed) Our first attempt was straightforward: re-run the scraper hit the same URLs fill the missing records It didn’t work. Why? Because: products no longer existed prices