Scraping Substack Newsletters: Content, Authors, and Subscriber Counts

Scraping Substack Newsletters: Content, Authors, and Subscriber Counts Substack has become the dominant newsletter platform with thousands of creators. Whether you are analyzing the newsletter landscape, researching competitors, or building a discovery tool, scraping Substack data provides valuable insights. What Data Can You Extract? Newsletter metadata : name, description, author info Subscriber counts : from leaderboards and public pages Post content : titles, excerpts, publication dates Categories and topics : how newsletters position themselves Setting Up pip install requests beautifulsoup4 pandas Scraping Newsletter Profiles Substack newsletters live at {name}.substack.com . Each has a public-facing page with metadata: import requests from bs4 import BeautifulSoup import json def scrape_substack_profile ( newsletter_slug ): url = f " https:// { newsletter_slug } .substack.com " headers = { " User-Agent " : " Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " } respons