A sales director described the scene to us like this. A prospect calls him, well advanced in their thinking, with a list of three suppliers in hand. The list came from an AI assistant. His company isn't on it. When he tried to understand why, no one at his firm could say whether the robots from OpenAI, Anthropic or Perplexity even had permission to read the site.
This is the most common situation we encounter in audits. The site is clean, classic SEO is sound, and yet the company is absent from generated answers, because some technical layer somewhere (an inherited robots.txt file, a default CDN setting, a firewall rule put in place two years ago against scraping) closes the door to assistants without anyone having decided it.
This guide covers the topic end to end. The first half is aimed at management, marketing and the CISO: what has changed, what these robots actually do, and which policy to adopt. The second half is aimed at the team that will implement: the step-by-step protocol, the files, the commands, the checks, then measurement and governance over time.
robots.txt and CDN or firewall configuration, and a Bing Webmaster Tools account (free). No knowledge of artificial intelligence is necessary. Command examples are given for Linux; they transpose easily.1. What has changed: the visit is delegated
For twenty years, the mechanics of the web were simple. An engine indexed your pages, a user typed a query, a list of links appeared, the user clicked, your page loaded. Each step left a measurable trace and each trace fed a dashboard.
Conversational assistants have inserted an intermediary into this chain. The user asks their question to ChatGPT, Claude, Perplexity or Copilot. The assistant decides alone whether it needs the web, launches its own searches, reads the pages it deems useful, synthesises, and responds. It sometimes cites its sources. The user rarely clicks. Your site may have been read, your argument may have weighed in the answer, but none of this appears in your audience analysis tool.
Three practical consequences follow.
The first is that traffic is no longer a good indicator of presence. A company can be cited a hundred times a day in generated answers and see only a handful of referred visits arrive. We'll return to this in the measurement section, as tools are just beginning to fill this gap.
The second is that access is negotiated robot by robot. Publishers have each set up, independently, a clear separation between their different web reading activities. This separation is the key to everything that follows.
The third is that blocking is silent. A robot that receives an error, a verification page or an excessive delay doesn't persist and doesn't warn anyone. The site disappears from answers without any signal going back.
2. The three permissions, and why they are independent
Each major publisher has the web read by several robots, identified by a distinct agent name in the robots.txt file. Despite different vocabularies, they cover three functions.
Indexing for search. A crawler continuously explores the web to build the index that the assistant queries when it needs sources. It's the equivalent of a classic search engine robot. At OpenAI, it's called OAI-SearchBot; at Anthropic, Claude-SearchBot; at Perplexity, PerplexityBot. Refusing it means disappearing from the list of candidate sources.
On-demand retrieval. When a user asks a question that requires reading a specific page, the assistant sends a robot to fetch that page, immediately. This is not automated exploration: there's a person behind each request. ChatGPT-User, Claude-User and Perplexity-User fulfil this role. Refusing it prevents the assistant from citing your page at the exact moment when someone is looking for what you offer.
Collection for training. A third robot gathers content that could be used to train future versions of models. GPTBot at OpenAI, ClaudeBot at Anthropic. Refusing it has no effect on your presence in today's answers. It's an intellectual property decision, not a visibility decision.
Two subtleties are worth noting before deciding anything.
The first concerns on-demand retrieval robots. Anthropic states that its three robots, including Claude-User, respect robots.txt. OpenAI removed at the end of 2025 the mention that ChatGPT-User complied with it, and Perplexity indicates that Perplexity-User generally doesn't apply it. The publishers' reasoning is that a person requested the page; the file addresses automata. You can dispute this reasoning, but you should know that a refusal in robots.txt isn't always enough to stop these agents, and that a network block will stop them, and the prospect behind them.
The second concerns Google. There is no separate robot for AI overviews and AI mode: it's Googlebot, the same as for classic search, that feeds everything. The Google-Extended token allows you to refuse the use of your content for training and grounding Gemini models outside of search, without touching your indexing. It doesn't remove you from generated answers in Google Search. This confusion is frequent, and it's at the heart of Cloudflare's rule change that we'll see later.
Three possible policies
In practice, the decision comes down to choosing between three positions. The vast majority of B2B companies find themselves in the second.
Which policy for your site?
| Visible without training | Fully open | |
|---|---|---|
| Search indexing | Authorised | Authorised |
| On-demand reading | Authorised | Authorised |
| Model training | Refused | Authorised |
| Presence in answers | Complete | Complete |
| Content reusable in future models | No (formal objection) | Yes |
| Copyright directive compatibility (TDM opt-out) | Machine-readable objection | No reservation expressed |
| Suits | B2B, publishers, proprietary content sites | Showcase sites, public documentation, open projects |
The third position, close everything, is justified for an exposed intranet, a transactional platform without editorial content, or a site whose content is the product itself. It should then be applied at the network level, not just in robots.txt, and assumed: the company won't exist in assistants' answers.
3. Where it blocks: the four layers
Here's the finding that surprises our contacts most. On the sites we audit, robots.txt is rarely the culprit. The blockage comes from a layer located before or after, that no one associates with AI visibility.
Layer 1, the CDN and application firewall. This is the most frequent case. Since July 2025, every new domain placed behind Cloudflare blocks identified AI robots by default, search robots included. On 1 July 2026, Cloudflare replaced this single switch with three categories, search, agent and training, separately adjustable, and announced new default settings that came into force on 15 September 2026: for newly added domains, new sites in an existing account and free accounts that haven't changed anything, the training and agent categories are blocked on pages displaying advertising, while search remains authorised. The tricky point is the treatment of multi-use robots: Googlebot, which serves both Google's search and AI functions, is subject to the most restrictive rule among those that concern it. A company that blocks training on an ad-funded site can therefore, unintentionally, slow down Google itself. Other CDNs and application firewalls (Akamai, Fastly, AWS WAF, Imperva) have managed lists of AI robots and comparable settings.
Layer 2, the host and server security. An indexing robot requests many pages in a short time. A rate limit that's too strict returns 429 errors; a tool like fail2ban or CrowdSec, configured with an automated browsing detection scenario, bans its IP address for several hours. Shared hosting providers sometimes add their own protections without informing the customer, precisely because robots consume CPU. The site is open on paper and closed in fact.
Layer 3, robots.txt. Classic errors are a User-agent: * rule with a broad Disallow, meant to stop content scrapers, that also stops any AI robot not explicitly named; a specific group declared for one robot but not for the others from the same publisher; a file corrected on the main domain but not on subdomains, when each host has its own. A useful rule to keep in mind: a robot only obeys the most specific group that names it, and then completely ignores the generic group.
Layer 4, the page itself. AI robots read HTML as it's served. A page whose content only appears after client-side JavaScript execution presents them with an empty shell. A forgotten noindex, content behind blocking consent or a login wall produces the same effect. This point is more for web teams than security, but it must be checked at the same time.
robots.txt as the only reliable opposition mechanism.4. The implementation protocol
The rest is aimed at the technical team and reads like a procedure. Each step produces a verifiable deliverable. Allow half a day for steps 1 to 4 on a simple site, a day if several domains or an enterprise CDN are involved.
4.1 Observe the existing situation
Start by establishing, with supporting evidence, what's happening today. Three checks suffice.
First read the robots.txt of each concerned host and identify the groups that touch AI robots.
for host in www.example.com blog.example.com docs.example.com; do
echo "== $host"
curl -sL "https://$host/robots.txt" | grep -iE -B1 -A3 'gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexity|google-extended|applebot|bytespider|ccbot|user-agent: \*'
doneThen search in access logs from the last thirty days for passages of these robots and especially the response codes they received. A 200 means the page was read; a 403, 429 or 503 means a layer refused.
zgrep -hiE 'OAI-SearchBot|ChatGPT-User|GPTBot|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Google-Extended|Applebot' /var/log/nginx/access.log* \
| awk '{ split($0, a, "\""); ua=a[6]; code=$9;
bot="other";
if (ua ~ /OAI-SearchBot/) bot="OAI-SearchBot";
else if (ua ~ /ChatGPT-User/) bot="ChatGPT-User";
else if (ua ~ /GPTBot/) bot="GPTBot";
else if (ua ~ /Claude-SearchBot/) bot="Claude-SearchBot";
else if (ua ~ /Claude-User/) bot="Claude-User";
else if (ua ~ /ClaudeBot/) bot="ClaudeBot";
else if (ua ~ /Perplexity-User/) bot="Perplexity-User";
else if (ua ~ /PerplexityBot/) bot="PerplexityBot";
n[bot" "code]++ }
END { for (k in n) print n[k], k }' | sort -k2,2 -k1,1nrOn a server behind Caddy with JSON-format logs, the same count is obtained with jq on the request.headers.User-Agent and status fields. If the site is behind Cloudflare, the origin server logs will never see robots blocked at the edge: open the dashboard, AI Crawl Control section (formerly AI Audit), which lists each robot, its requests and what happened to them.
Finally, if a behavioural firewall runs on the server, list its recent decisions. With CrowdSec:
sudo cscli decisions list -o json | jq -r '.[] | [.value, .scenario, .duration] | @tsv' | head -50
sudo cscli alerts list --scenario crowdsecurity/http-crawl-non_staticsRecord the result in a four-column table: robot, last passage, dominant response code, responsible layer. This is the initial state you'll compare after intervention.
4.2 Write the policy and get it signed
Before touching any file, formalise the decision. It fits on one page and engages three functions: marketing or communications, which carries the visibility objective; security, which carries the blocking rules and robot verification; legal or management, which decides on training.
The legal aspect isn't decorative. The European copyright directive allows a rightholder to object to text and data mining for commercial purposes, provided this objection is expressed in a machine-readable manner; robots.txt is today the most recognised vehicle for this reservation. The European AI regulation, whose obligations for general-purpose models have applied since 2 August 2026, requires model providers to have a copyright compliance policy, including these reservations. Refusing training in robots.txt therefore has real legal scope, and tacitly authorising it does too.
The decision sheet contains at minimum: the policy chosen among the three in chapter 2; the list of covered hosts; exceptions by directory (customer area, internal search, account pages); the configuration owner; the date of the next review. Once signed, it becomes the reference against which each technical layer is verified.
4.3 Write robots.txt
Here's a complete file for the "visible without training" policy, which suits most business sites. Adapt excluded directories to your structure.
# AI visibility policy · revised 2026-09-17 · owner: web team
# Search and on-demand reading authorised, training refused.
# --- OpenAI ---
User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /search
User-agent: ChatGPT-User
Allow: /
Disallow: /account/
User-agent: GPTBot
Disallow: /
# --- Anthropic ---
User-agent: Claude-SearchBot
Allow: /
Disallow: /account/
Disallow: /search
User-agent: Claude-User
Allow: /
Disallow: /account/
User-agent: ClaudeBot
Disallow: /
# --- Perplexity ---
User-agent: PerplexityBot
Allow: /
Disallow: /account/
User-agent: Perplexity-User
Allow: /
# --- Google: Gemini training token (doesn't affect search or AI overviews) ---
User-agent: Google-Extended
Disallow: /
# --- Apple ---
User-agent: Applebot-Extended
Disallow: /
# --- Massive collection without visibility return ---
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# --- All other robots ---
User-agent: *
Allow: /
Disallow: /account/
Disallow: /search
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlThree form rules avoid bad surprises. Each authorised robot must have its own group, even if the rules are identical to those of the generic group, because a robot named in a specific group applies only that one. The file must exist on each subdomain, with its own rules. And the consideration delay must be anticipated: OpenAI indicates approximately twenty-four hours for its systems to account for a modification; other publishers don't commit to a duration, allow a few days.
Anthropic indicates respecting the non-standard Crawl-delay directive; if your server suffers under the load of a legitimate robot, it's this lever you should use rather than a network block that would then prevent it from reading your instructions.
4.4 Open infrastructure layers
This is the step that requires the most coordination with security, and it's the one that most often unblocks the situation.
On Cloudflare. Open the zone, Security section then Bots or AI Crawl Control depending on your dashboard version. Since 1 July 2026, you have three independent settings: search, agent, training. Set search to authorised, agent to authorised if you want to be cited on demand, and training to blocked if your policy provides for it. Then check the nominative list of robots to confirm that OAI-SearchBot, Claude-SearchBot and PerplexityBot don't appear in a blocking rule inherited from the old single switch. If your site displays advertising, reread the treatment reserved for Googlebot: with training blocking active, the most restrictive rule applies to this multi-use robot, and you must use the explicit exception offered by Cloudflare to preserve your presence in Google search. Other CDNs offer an equivalent under the names of bot management or verified robot lists.
On a behavioural firewall. The objective is to exclude legitimate robots from automated browsing detection scenarios without opening the door to imposters. For OpenAI, whose IP ranges are published, the safest approach is an address-based allow list, automatically refreshed.
# Retrieves IP ranges published by OpenAI and puts them in CrowdSec allow list
for bot in gptbot searchbot chatgpt-user; do
curl -s "https://openai.com/${bot}.json" | jq -r '.prefixes[].ipv4Prefix // empty'
done | sort -u > /etc/crowdsec/openai-prefixes.txt
sudo cscli allowlists create ai-robots -d "AI robots verified by publisher IP range" 2>/dev/null || true
while read -r cidr; do
sudo cscli allowlists add ai-robots "$cidr" -d "OpenAI"
done < /etc/crowdsec/openai-prefixes.txtFor Anthropic, which doesn't publish ranges, add a whitelist by agent name expression, accepting that it's spoofable, and compensate with a reasonable rate limit rather than a ban. Place this refresh in a weekly task: ranges evolve.
On the web server. Verify that the rate limit doesn't apply to verified robots, or that it lets through a hundred requests per minute, which covers normal indexing crawl. Verify that no security rule (mod_security, OWASP rules) rejects these agents based solely on their name, which happens with old rulesets that classify "bot" as suspect.
At the host. If you're on a shared offer or managed platform, request in writing the list of active anti-robot protections and the exclusion of agents in your policy. Keep the response with the decision sheet.
4.5 Make content readable
Once the door is open, what's behind must be usable by a hurried automatic reader.
The main content of each page must be present in the served HTML, without depending on client-side JavaScript execution. Modern frameworks all allow this via server-side rendering or static generation; verify it by retrieving a page with curl and searching for your first paragraph in the raw response.
The sitemap must be up to date, referenced in robots.txt, and reflect the actual last modification date of each page. AI indexing robots use it to prioritise. Canonical tags must be consistent, publication and update dates visible, the author identified. Structured data (organisation, article, FAQ, product) help engines understand what the page is about; Google specifies that no specific markup is required for its AI functions, but clean markup remains an advantage for all.
The llms.txt file, proposed as a model-readable index at the site root, is adopted by some tools and ignored by others; Google indicates not needing it. It costs ten minutes and doesn't hurt. Consider it optional. Finally, for Copilot, register the site on Bing Webmaster Tools and activate IndexNow to signal your updates without waiting for the robot's next pass.
4.6 Measure
This is the youngest link in the chain, and it's progressing quickly. Three sources complement each other.
Logs, with the 4.1 script relaunched each month, give the number of verified passages per robot and the list of pages read on demand. This second list is valuable: a page retrieved by ChatGPT-User or Claude-User is a page that served to answer someone. It tells you which content works for you.
Bing Webmaster Tools offers since February 2026 an AI Performance report, enriched in June 2026 with views by intent, by topic, citation share and comparison. It indicates how many times your pages were cited in Copilot and Bing experiences responses, which pages, and especially the grounding queries: the internal reformulations that the assistant generated to find your content. It's the first official data of this type published by a major publisher, and it only measures visibility, not clicks. On the Google side, AI overviews are merged in the Search Console Performance report without isolation; a dedicated report has been in testing since June 2026 on a restricted scope. Don't count on it short term.
The question panel is the only measure that covers all four assistants. Build a list of twenty to forty questions your customers actually ask (the sales team knows them), ask them monthly to ChatGPT, Claude, Perplexity and Copilot in private browsing, and note for each whether your company is cited, at what position, and whether what's said is accurate. A spreadsheet suffices at first; specialised tools automate the exercise when volume justifies it.
4.7 Govern over time
Configuration degrades by itself. A new subdomain is born without robots.txt, a site redesign switches back to client rendering, a security contractor resets the firewall, the CDN changes its defaults, a publisher renames a robot. Three safeguards suffice.
An automated test, launched by continuous integration on each deployment, that verifies that the robots.txt of each host indeed contains the policy groups and that the home responds with 200 to a request bearing the agent name of each authorised robot. A twenty-minute monthly review around the measurement table. A quarterly review of the decision sheet, with rereading of publisher documentation, whose robot names and rules change several times a year.
5. What we apply on nAIvigate
This site runs on a dedicated server, behind Caddy, with CrowdSec as behavioural firewall and no CDN. In other words, layer 1 doesn't exist for us but layer 2 is active, and it's what we monitor. Our monthly protocol consists of four commands and a table.
We extract from Caddy's JSON logs the passages of each AI robot with their response code; we compare the list of CrowdSec decisions to these addresses; we relaunch the robots.txt test robot by robot on our three witness URLs; we read Bing's AI Performance report and our thirty-question panel. Everything feeds a line in a table going back to the site's opening.
The first complete pass, done on 17 September 2026 on seven days of logs, gave this. No blocking: each AI robot received 200s, and no publisher address appeared in CrowdSec bans. But the visit distribution said something other than what we supposed. PerplexityBot had passed 1,557 times, Meta's collector 410 times, Amazon's 384 times, Bingbot 126 times. Googlebot, 11 times for over five hundred pages. OAI-SearchBot and ClaudeBot, four times each. And not a single on-demand read, by any assistant, in a week. The robots.txt file, generated by the framework, only named training robots and left others on the generic rule; it also forbade, via an inherited Disallow: /_next/, the directory of stylesheets and scripts that Google must load to render the page.
Two lessons emerged. Blocking wasn't our problem, crawl priority was: a three-month-old site isn't yet in OpenAI's or Anthropic's queue, and nearly two-thirds of AI robot traffic went to two collectors that return no visibility. And the most costly error wasn't in the AI column of the table, but in that /_next/ that had penalised classic SEO since launch. The file was rewritten that same day to name the thirteen robots from the chapter 2 matrix and reopen static resources; registration with Bing Webmaster Tools and Search Console became the month's two priority actions. The blockages that matter never appear in the file everyone looks at, they appear in the logs no one reads.
6. The errors we see most often
Blocking GPTBot thinking it removes you from ChatGPT. The result is opposite to the intention: content no longer serves training, which was perhaps wanted, but the site remains entirely visible in answers. If the objective was not to appear there, it's OAI-SearchBot and ChatGPT-User that needed treating, and at the network level for the second.
Correcting robots.txt and stopping there. On Cloudflare, the inherited blocking rule continues to apply at the edge; the robot never reads the corrected file.
Waiting for an effect on traffic. The right metric is citation, not visit. A company well positioned in answers can see its visits fall while gaining qualified appointments, because the prospect arrives later in the journey and better informed.
Producing content for robots. Assistants cite what clearly answers a precise question, with dated facts, figures and an identifiable author. Generic content mass-produced is read, compared, and discarded in favour of a clearer source. A single complete reference guide is worth more than fifty lukewarm pages.
Forgetting subdomains. Documentation, blog, press area each have their robots.txt and CDN configuration. Main domain open and documentation closed is a frequent combination, and it's often the documentation you'd like cited.
7. Test your understanding
Your company wants to be cited by ChatGPT but refuses that its content serve to train OpenAI models. Which robots.txt configuration is correct?
📚Glossary (expand)
AI robot (AI crawler): an assistant publisher's program that reads web pages. Each publisher operates several, distinguished by their function.
Agent name (user-agent): string by which a robot identifies itself in each request and in robots.txt. Falsifiable, hence verification by IP address.
robots.txt: text file at the root of each host that indicates, agent by agent, allowed or refused paths. Respected by legitimate robots, not by others. Specified by RFC 9309.
Specific group: set of rules addressed to a named robot. A named robot applies only its group and ignores the generic * group.
Indexing: continuous construction of a page index that the assistant queries to find sources.
On-demand reading (user-fetch): retrieval of a specific page triggered by a user's question. Some publishers consider that robots.txt doesn't apply to it.
Training: collection of content intended for learning future versions of a model. No effect on visibility in current answers.
Grounding query: internal reformulation generated by an assistant to search for sources before responding. Exposed in Bing's AI Performance report.
Google-Extended: control token read by Googlebot to refuse use of content in Gemini outside search. No effect on indexing or AI overviews.
AI overviews and AI mode: generated answers displayed in Google Search, fed by Googlebot and the search index.
CDN (content delivery network): intermediary layer between the user and your server, that caches and filters. Cloudflare, Akamai, Fastly. Can block robots before they reach your server.
WAF (web application firewall): HTTP request filter according to rules. Can reject a robot on its name alone.
Behavioural firewall: tool (CrowdSec, fail2ban) that bans an IP address after detecting suspicious behaviour, for example rapid browsing of many pages.
Rate limit: request ceiling per time unit, beyond which the server returns a 429 error.
Server-side rendering: production of complete HTML by the server, readable without executing JavaScript. Necessary for robots to see content.
llms.txt: proposed index file at the site root, intended for models. Partial adoption, not required by Google.
IndexNow: page update notification protocol, used by Bing.
TDM opt-out: reservation expressed by a rightholder against text and data mining for commercial purposes, provided by the European copyright directive. Must be machine-readable.
GEO (Generative Engine Optimisation): set of practices aimed at presence and citation in generated answers. This guide covers the access layer, prerequisite to everything else.
Frequently asked questions
How long before seeing an effect? One day for OpenAI after robots.txt modification, a few days for others, then the time for robots to revisit your pages. Allow two to four weeks before the Bing report and your panel move.
Should you authorise training to be better cited? No. No publisher conditions visibility in its answers on training authorisation, and all have separated robots precisely for this reason.
Does an GDPR consent wall block robots? If content is present in HTML behind the banner, no. If the banner retains content until click, yes. Verify with curl.
Perplexity-User ignores robots.txt, what to do if you don't want to be read? The only means is a network block, which will also block the user behind. It's a decision to assume with full knowledge, and to limit to truly sensitive directories.
Is llms.txt mandatory? No. It's read by some tools and ignored by others. It in no way replaces correct robots.txt and readable content.
Who should own this topic in the company? Visibility is marketing's responsibility, blocking is security's, training is legal's. The decision sheet exists so these three functions sign the same document instead of contradicting each other in three different configurations.
To go further
The nAIvigate Trust Index documents, publisher by publisher, what each AI provider does with the data it receives; it's the natural complement to this guide for the inverse question, that of data you entrust to these same publishers. Our AI terms glossary for IT and security leaders covers this guide's concepts in their broader context.
If you want to know in half a day where your site really sits, with the four-layer table, corrected robots.txt and ready-to-use measurement protocol, it's the type of finding an AI Radar produces.