Scraping Lobbying Disclosure Data: Following the Money

Introduction Lobbying disclosure data reveals who is spending money to influence legislation. In the US alone, over $4 billion is spent annually on lobbying. This data is publicly available through government databases but often buried in hard-to-access formats. In this tutorial, we'll build scrapers to collect and analyze lobbying disclosure data. Setup import requests from bs4 import BeautifulSoup import pandas as pd import json import time from datetime import datetime import sqlite3 # For handling government sites with anti-bot protection # Get your API key: https://www.scraperapi.com?fp_ref=the52 SCRAPER_API_KEY = " your_key_here " BASE_URL = " http://api.scraperapi.com " Scraping the Senate Lobbying Database The US Senate maintains a searchable lobbying disclosure database: def search_senate_lobbying ( registrant = None , client = None , year = 2026 ): """ Search the Senate Lobbying Disclosure database. """ url = " https://lda.senate.gov/api/v1/filings/ " params = { " filing_year