A number of university libraries across North America report a sharp rise in traffic generated by automated bots systematically harvesting their digital collections. These crawlers, often linked to AI companies seeking training data, target specialized archives, digitized manuscripts and heritage collections that are typically accessed by a limited number of researchers.
The issue creates a concrete infrastructure problem: servers built for moderate academic use struggle to absorb sudden, sometimes massive spikes in automated requests. Several librarians cited in reporting describe slowdowns and even service outages that affect legitimate users precisely when they try to reach these resources. Some institutions have responded by deploying firewalls, human verification systems or rate limiting, measures that run counter to their traditional mission of open access to knowledge.
The pattern echoes a broader trend already seen on other open infrastructures, such as open-source code repositories and Wikipedia, where a surge in AI-related scraping has forced maintainers to rethink access policies. What sets university libraries apart is their role as custodians of rare, often publicly funded collections, which makes the question of restricting access particularly sensitive from both an ethical and institutional standpoint.
Beyond the technical strain, the episode reignites debate over the use of protected or heritage materials to train commercial models without compensation or explicit consent from the institutions involved. Several libraries are reportedly considering stricter robots.txt protocols or specific licensing arrangements, though such measures remain fragile against bots that frequently ignore these conventions. The situation highlights a growing tension between libraries' historic commitment to openness and the need to shield their resources from unregulated commercial exploitation.