Back to articles
How to Scrape DuckDuckGo Search Results (No API Key, No Blocks)

How to Scrape DuckDuckGo Search Results (No API Key, No Blocks)

via Dev.to WebdevАлексей Спинов

DuckDuckGo is the easiest search engine to scrape. No blocks, no CAPTCHAs. DuckDuckGo Instant Answer API (Free) async function ddgSearch ( query ) { const url = `https://api.duckduckgo.com/?q= ${ encodeURIComponent ( query )} &format=json&no_html=1` ; const res = await fetch ( url ); const data = await res . json (); return { abstract : data . Abstract , abstractSource : data . AbstractSource , abstractURL : data . AbstractURL , relatedTopics : ( data . RelatedTopics || []). map ( t => ({ text : t . Text , url : t . FirstURL })), answer : data . Answer }; } HTML Scraping (Full Results) DDG HTML version is scrapable without anti-bot: async function ddgFullSearch ( query ) { const url = `https://html.duckduckgo.com/html/?q= ${ encodeURIComponent ( query )} ` ; const res = await fetch ( url ); const $ = cheerio . load ( await res . text ()); return $ ( " .result__title " ). map (( i , el ) => ({ title : $ ( el ). text (). trim (), url : $ ( el ). find ( " a " ). attr ( " href " ) })). get

Continue reading on Dev.to Webdev

Opens in a new tab

Read Full Article
2 views

Related Articles