Running Headless Chrome at Scale: Production Lessons from Millions of Renders.

Running Headless Chrome at Scale: Production Lessons from Millions of Renders. Anyone who has crawled websites at scale knows that a simple HTML fetch is not enough. Too many websites client-side render their content, which means the HTML you get from a plain HTTP request is an empty shell. To get the actual content, you need a browser. In practice, that means headless Chrome. Running one instance of headless Chrome is straightforward. Running it at scale - processing millions or tens of millions of pages on a regular basis - is a different problem entirely. At EdgeComet, we run hundreds of Chrome instances across our servers. We've spent years fine-tuning and managing Chrome page rendering on a scale. This article covers managing Chrome itself. We will omit the process of parsing and storing data. The focus here is on the hard part: keeping Chrome alive, responsive, and fast when you push it far beyond what it was designed for. Do You Even Need Chrome? Wherever you can avoid using hea