How AI agents browse the web (and why your site might be invisible to them)

How AI agents browse the web (and why your site might be invisible to them) Most sites are built for humans sitting in front of a browser. That's fine, except AI agents aren't humans, and they don't use browsers. When an agent needs to interact with your product, it sends an HTTP request, reads the response, and tries to extract something useful. No JavaScript engine, no mouse clicks, no scroll events. If your content only exists after a React component mounts, the agent sees an empty shell. If your site is behind Cloudflare's bot challenge, the agent never gets past the waiting room. How an agent actually 'sees' a website The sequence is roughly this: HTTP GET to a URL, parse the HTML or JSON response, check for an llms.txt file at the root, and look for any documented API it can call. That's the whole pipeline. llms.txt is a simple convention, similar in spirit to robots.txt , that gives agents a curated map of what your site contains and how to use it. If you haven't heard of it, th