Scraping OpenAlex and Semantic Scholar for Research Intelligence

Academic research intelligence is a growing field. Whether you're tracking emerging technologies, monitoring competitors' R&D, or building citation networks — OpenAlex and Semantic Scholar are the two largest open databases of scholarly work. Here's how to extract intelligence from both. OpenAlex vs Semantic Scholar OpenAlex : 250M+ works, fully open API, no auth required, covers all disciplines. Run by the nonprofit OurResearch. Semantic Scholar : 200M+ papers, AI2-backed, excellent AI/ML/CS coverage, free API with rate limits. Querying OpenAlex import requests import time class OpenAlexClient : BASE_URL = " https://api.openalex.org " def __init__ ( self , email = None ): self . session = requests . Session () if email : self . session . params = { " mailto " : email } # Polite pool def search_works ( self , query , filters = None , per_page = 50 , pages = 3 ): all_results = [] for page in range ( 1 , pages + 1 ): params = { " search " : query , " per_page " : per_page , " page " : pa