What OpenAI documents about its crawlers
OpenAI publishes a short crawler reference with separate settings for each bot. Treating them as one switch is a common mistake. A fourth agent, OAI-AdsBot, only visits landing pages submitted as ChatGPT ads, so it does not affect organic visibility.
OAI-SearchBot
Surfaces websites in ChatGPT's search features. A site that opts out is not shown in ChatGPT search answers, though it may still appear as a navigational link.
GPTBot
Collects content that may go into training OpenAI's generative AI foundation models. Disallowing it signals that your content should not be used for training.
ChatGPT-User
Handles certain user-initiated actions, such as visiting a page when someone asks about it. OpenAI notes robots.txt rules may not apply, because the user initiates the request.
Allowing the right crawler in robots.txt
OpenAI says each setting is independent. A site can allow OAI-SearchBot so it can appear in ChatGPT search while disallowing GPTBot to opt out of training. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing its published IP ranges.
A robots.txt file is plain text, so the change is small. The risk lies in broad rules written years ago, or in a plugin that adds a blanket block without telling anyone. We read the live file, not the one you remember, and list every rule that touches an AI crawler.
OpenAI says changes can take about 24 hours to reach search. For that reason we make one change at a time and wait before testing.
The CDN and firewall layer
A permissive robots.txt does not help if a firewall rejects the request first. Many sites sit behind a CDN or web application firewall, and its bot settings can stop a crawler before it reads robots.txt.
Cloudflare sorts AI traffic into three behaviours, Search, Agent and Training, and lets every customer set each one to allow, block, or block on pages that display ads. For new domains, the defaults since 15 September 2026 block Training and Agent bots on pages with ads and allow Search. Cloudflare's 15 September post says existing settings were migrated to the new controls, so most existing sites needed no action.
How the main AI crawlers compare
Other assistants publish their own crawler details. This table summarises what each provider says, as we read it on 23 October 2026.
| Crawler | Documented purpose | Robots.txt position |
|---|---|---|
| OAI-SearchBot (OpenAI) | Surfaces sites in ChatGPT search | Honoured for search opt-outs |
| ClaudeBot (Anthropic) | Collects content that may be used for training | Honoured; Crawl-delay supported |
| Claude-SearchBot (Anthropic) | Analyses content to improve search result quality | Honoured; blocking may reduce search visibility |
| Claude-User (Anthropic) | Retrieves pages when a Claude user poses a question that needs them | Honoured; blocking may reduce visibility in user-directed search |
| PerplexityBot (Perplexity) | Surfaces and links sites in Perplexity results | Allowing it is recommended; not used to train foundation models |
| Perplexity-User (Perplexity) | Fetches a page when a user's question needs it | Generally ignores robots.txt, as the user requested the fetch |
Making pages worth fetching
Access only gets a crawler to the door. These steps affect what it finds.
- Say what you do plainly: A page that names the service, the area served and the conditions is easier to describe correctly than a page of slogans.
- Lead with the answer: Put the direct reply to a common question near the top, then add detail below it.
- Keep details current: Out-of-date prices, hours or service lists are the usual source of wrong answers about a business.
- Match your listings: Align the name, area and contact methods on your site with your profiles and directories.
About llms.txt
llms.txt is a proposal, first made in 2024, for a Markdown file that gives AI tools a curated overview of a site. Its site now describes a second version. It is not a formal standard.
What we can say
- The proposal exists and is documented at llmstxt.org.
- Google states that Search does not use llms.txt or similar files.
- Adding one is low effort and does no harm.
What we cannot say
- That OpenAI uses it to choose ChatGPT sources; its crawler page does not mention it.
- That Anthropic or Perplexity use it; their crawler pages do not mention it either.
- That it improves visibility anywhere.
How we test and report
ChatGPT does not give site owners a reporting dashboard of the kind Search Console offers, so we test by hand.
- Write the question setTen to twenty questions, phrased as your customers would ask them, covering services, comparisons and local needs.
- Ask and recordWe run each question, save the answer, note the sources shown, and record the date and the exact wording.
- Check your server logsIf you can share them, access logs show whether the crawlers listed above are reaching your pages.
- Repeat laterWe rerun the set on a schedule, since a single answer says little and results vary between runs.
What to prepare before an access check
A short list of items makes the review faster and keeps guesses out of it. None of them needs technical skill to gather.
- The live robots.txt: Open yourdomain.com.au/robots.txt in a browser and save the text, since plugins and hosts can change it without notice.
- CDN or firewall access: Know who manages Cloudflare, your host or your security plugin, and whether a bot-blocking toggle was ever switched on.
- Your reasons for blocking: Decide whether you object to training use, to answer use, or both, because the crawlers are separate and so are the choices.
- Ten real customer questions: Use the wording customers use on the phone or in enquiry emails, not the wording from your own service pages.
- Server logs, if available: Ask your host for a week of access logs so the user agents named above can be searched for by name.
Where this fits
ChatGPT work builds on technical SEO and the page work in answer engine optimisation. For the wider question of how AI tools describe your business, read generative engine optimisation. Everything in this topic is mapped on the AI search page.
Frequently asked questions
How do I get my business to show up in ChatGPT?
OpenAI says to allow OAI-SearchBot in robots.txt and allow its published IP ranges. Beyond access, publish clear, accurate pages. OpenAI does not publish how ChatGPT picks the sources it shows, so we can improve your chances of being readable but cannot promise a mention.
Should I block GPTBot?
That is your decision. OpenAI says disallowing GPTBot signals your content should not be used to train its models, and that each crawler setting is independent. You can block GPTBot and still allow OAI-SearchBot so your site can appear in ChatGPT search.
Does robots.txt control ChatGPT-User?
OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person initiates those visits, for example by asking about a page. It also says ChatGPT-User does not decide whether content appears in search. Use OAI-SearchBot for search opt-outs.
Is llms.txt worth adding?
It is a proposed convention, not a confirmed signal. Google says Search does not use it, and the OpenAI, Anthropic and Perplexity crawler pages we read do not say they do. It costs little to publish, but do not expect a measurable effect.
Can Cloudflare block ChatGPT from my site?
Yes. Cloudflare's bot settings can stop a crawler before it sees robots.txt. Its Agent category covers chat fetch bots such as ChatGPT-User, and its Training category covers GPTBot. Look at the Search, Agent and Training settings in your dashboard and confirm which one matches the crawler you want to allow, rather than assuming a default.
Does Claude or Perplexity use the same rules?
No. Anthropic lists ClaudeBot, Claude-User and Claude-SearchBot, each with its own purpose, and Perplexity lists PerplexityBot and Perplexity-User. Each needs its own robots.txt and firewall decision. We cover them in the same access review.
How do I know if ChatGPT mentions me?
There is no official report we can read. We ask a fixed set of customer questions, record answers, sources and dates, and repeat the set later. It is manual and the results vary, but it shows real wording and any factual errors.
Sources and further reading
- OpenAI: Overview of OpenAI crawlers: OAI-SearchBot, GPTBot and ChatGPT-User purposes; robots.txt behaviour; about 24 hours for changes.
- Anthropic: Does Anthropic crawl data from the web: ClaudeBot, Claude-User, Claude-SearchBot; robots.txt and Crawl-delay.
- Perplexity: Perplexity crawlers: PerplexityBot and Perplexity-User; IP ranges; WAF guidance; robots.txt behaviour.
- Cloudflare Docs: Block AI Bots: Search, Agent and Training behaviours; allow, block, or block on pages with ads; new-domain defaults from 15 September 2026.
- Cloudflare Blog: Have it both ways, stay discoverable in search while disallowing AI training: Published 15 September 2026. Block applies to mixed-use crawlers such as Googlebot; Disallow AI Training keeps them allowed; training-only crawlers of OpenAI and Anthropic.
- Cloudflare Changelog: New options to manage AI traffic: 1 July 2026 release of the three categories for all plans, including Free.
- llms.txt: A proposal to provide information to help LLMs use websites: llms.txt is a proposal, now at version 2; not a formal standard.
- Google: Optimizing for generative AI features on Google Search: Google Search does not use llms.txt.