I built 59 scrapers for US government data and most of them were a waste of time

I've spent the last few months building web scrapers for US government databases. Secretary of State business filings, SEC EDGAR, FDA clearances, OSHA inspections, court records, contractor licenses -- 59 of them total, all in Go, all deployed on Apify. Most of them were a waste of my time. Here's what I actually learned. The government data ecosystem is weirdly split There are two kinds of government data sources and the experience of scraping them is completely different. The good ones have real REST APIs with JSON responses, pagination, rate limits, and actual documentation. SEC EDGAR, NIH RePORTER, ClinicalTrials.gov, the NVD CVE database, Regulations.gov -- these are legit. You write a Go HTTP client, handle pagination, and you're done in a day. Some of them are genuinely well-designed APIs that private companies should be embarrassed by. The bad ones are ASP.NET forms from 2004 with ViewState tokens, session cookies that expire every 3 minutes, and search results that render clie