What the Scrapy Maintainer Thinks About AI-Generated Scrapers

I sat down with Adrian Chaves, one of the lead Scrapy maintainers, who also works at Zyte, to ask him the questions I've been chewing on since Zyte launched Web Scraping Copilot: what happens when an LLM writes your spider(the web scraping code)? What gets easier? What doesn't change? His answers surprised me. A few highlights: On vibe coding: Adrian has thoughts about developers treating scraper generation as a black box, and why Scrapy's design philosophy matters more, not less, when an LLM is writing the code. The bottleneck isn't what you think. He argues the hard part of scraping in 2026 isn't writing code. It's reading pages. And that's the part LLMs still struggle with. What "good design meeting the future halfway" means. Why frameworks like Scrapy that were built for humans are turning out to be the best frameworks for AI agents too. Where LLMs actually help. The concrete places where AI makes a scraper developer's life better, and where it just adds complexity. Full conversati