Short answer: Treat AI crawler rules as separate decisions. Allow the bots that support search visibility, decide separately if training access is acceptable, and keep Googlebot available when you want Google Search and its AI features to crawl the page. Test the final file and monitor logs after publishing.
Start by separating the goals
A robots.txt policy is easier to manage when it answers three different questions. Do you want a page to be found in a search product? Do you want its content to be available for model training? Do you want a user-triggered fetch to work when someone asks an assistant to open a URL? One rule should not be treated as an answer to all three questions.
OpenAI documents OAI-SearchBot and GPTBot as independent controls. OAI-SearchBot supports inclusion in ChatGPT search, while GPTBot is used for crawling content that may be used to train OpenAI foundation models. The current OpenAI crawler guidance explains that a site can allow one and disallow the other.
For provider-specific planning, compare Google Gemini Optimization with Perplexity AI visibility so the crawl policy follows the platforms the audience actually uses.
What each crawler decision means
| Crawler | Main purpose | Decision to make |
|---|---|---|
| OAI-SearchBot | ChatGPT search discovery | Allow it when ChatGPT search visibility is part of the plan |
| GPTBot | Potential training use for OpenAI foundation models | Allow or disallow based on your content and licensing position |
| Googlebot | Google Search crawling and indexing | Keep it available for pages you want in Google Search |
| Google-Extended | Google's separate control for some AI training and grounding systems | Review it separately from Googlebot |
| PerplexityBot | Perplexity search crawling | Use a specific rule if you need a different policy for Perplexity |
These labels describe the purpose of the user agent, not a ranking promise. An allowed crawler can access a page, but access does not guarantee a citation, a position, or inclusion in an answer. A blocked crawler may also leave a URL known through other sources, so robots.txt is not a complete removal mechanism.
Write a policy that matches the decision
Rules are grouped by user agent. A simple policy can allow a search crawler while restricting a training crawler on a private path:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /private/
User-agent: Googlebot
Allow: /Use the actual paths and policy approved for your site. Do not paste a sample into production without checking the effect of each group. A broad User-agent: * block can catch crawlers you intended to allow, while a rule aimed at one bot does not automatically control another bot with a different name.
Keep the file at the root of the host it governs. Check the protocol, hostname, and port, because a robots.txt file on one host does not automatically set rules for a different subdomain. Make the file plain UTF-8 text, keep directives readable, and include a sitemap URL when that is part of your technical setup.
Know what robots.txt can and cannot do
Robots.txt controls crawl access. It does not function like a page-level noindex instruction, a licensing contract, or a deletion request. If the aim is to prevent an already public page from appearing in a search index, use the appropriate page or response controls and confirm that the crawler can reach them. Google's guide to AI features and website controls explains that Googlebot access governs Search eligibility, while Google-Extended is handled separately for some other Google systems.
Also account for infrastructure. A web application firewall, CDN rule, authentication wall, rate limit, or JavaScript challenge can block a crawler even when robots.txt says Allow. Review server logs and test the response delivered to the relevant user agent instead of assuming that the text file tells the whole story.
A practical review checklist
- List the business objective: Decide which search products, training uses, and direct fetches matter to the site.
- Map the user agents: Write a separate policy for each provider and do not combine names because they sound similar.
- Check rule scope: Confirm that the intended Allow and Disallow paths are under the correct group.
- Check delivery: Request the root robots.txt file from the public host and inspect the HTTP status, encoding, and redirects.
- Check logs: Look for the crawler name, response status, blocked paths, and unexpected traffic after deployment.
- Record the date: Provider behavior and user-agent names change, so keep a short decision note for the next review.
A sensible default for a visibility-focused brand
A brand that wants to be found in AI search will usually keep Googlebot available and evaluate OAI-SearchBot and PerplexityBot as search access decisions. GPTBot and Google-Extended deserve a separate discussion about training, rights, and the value of contributing content. No universal answer; the correct policy depends on the site's commercial model and the terms it can support.
Common mistakes to avoid
- Blocking every AI-related user agent when the goal is only to limit training.
- Assuming GPTBot controls ChatGPT search visibility. OpenAI identifies OAI-SearchBot as the search control.
- Using robots.txt to hide confidential information that should be protected by authentication.
- Changing rules without checking the canonical host and subdomains.
- Expecting a crawl rule to remove content that has already been copied or indexed elsewhere.
Frequently asked questions
Should GPTBot and OAI-SearchBot have the same rule?
No. They serve different purposes. Keep the rules aligned only if your policy treats search access and training access the same way.
Can I allow Googlebot and block Google-Extended?
Google describes these as separate controls. Review the current Google documentation, test the resulting behavior, and preserve Googlebot access when the page must remain eligible for Google Search.
How often should the file be reviewed?
Review it after a provider changes its crawler documentation, after a hosting or CDN migration, and whenever server logs show an unexpected bot or status pattern. A small change can affect a large set of URLs.