Web Scraping for Claude Code: Strip Nav, Keep Content

via Dev.to Webdevjidong4h ago

Claude Code can fetch web pages. But when it does, it gets the full HTML — navigation bars, footer links, cookie banners, script tags, SVG icons, meta tags. On a typical documentation page, the actual content is 30% of the HTML. The other 70% is page structure, navigation, and decoration. Before: Raw HTML Fetch <!DOCTYPE html> <html> <head> <meta charset= "utf-8" > <meta name= "viewport" content= "width=device-width" > <title> API Reference - MyLib </title> <link rel= "stylesheet" href= "/styles.css" > <script src= "/analytics.js" ></script> </head> <body> <nav class= "sidebar" > <a href= "/" > Home </a> <a href= "/docs" > Docs </a> <a href= "/api" > API </a>  </nav> <main> <h1> createUser() </h1> <p> Creates a new user with the given parameters. </p> <pre><code> const user = await createUser({ name, email }) </code></pre> </main> <footer>  </footer> </body> </html> Your AI needs the <main> content. It gets everything. After: Through

Continue reading on Dev.to Webdev

Opens in a new tab

Read Full Article

7 views