A growing number of publishers and online platforms are tightening access policies as AI agents capable of browsing, reading and acting on the web on behalf of users become more common. According to a report from The Verge, this trend is accelerating as such agents, built to make purchases, compare prices or extract information, bypass the usual pathways designed for human visitors, including advertising pages and sign-up forms.
This shift follows years of friction between content publishers and the crawlers used by AI companies to gather training data. While blocking bots through robots.txt files or services like Cloudflare was already widespread, the new development is that sites are now specifically targeting so-called agentic systems, which interact with websites in real time rather than simply harvesting data for later model training.
Publishers' motivations vary. Many want to protect advertising revenue, which typically depends on human browsing behavior, while others aim to retain control over the user experience and prevent automated agents from skewing traffic analytics or bypassing payment flows. E-commerce sites in particular worry that automated shopping agents could complete transactions without going through channels that generate commissions or usable customer data.
If this trend continues, it could reshape parts of the web's underlying economics. Companies building AI agents may increasingly need to strike commercial agreements with publishers, similar to the content licensing deals some model makers have already signed. Otherwise, the open web risks splitting into zones accessible to agents and zones that remain closed off, a challenge that affects large media organizations and small merchants alike.