# Robots.txt for AI Search: How to Allow Search Visibility Without Making One Blanket Bot Decision Source: https://enginely.ai/blog/robots-txt-for-ai-search-how-to-allow-search-visibility-without-making-one-blanket-bot-decision/ > Build a robots.txt policy for AI search without confusing search visibility, model training and user fetchers. Includes practical examples and checks. August 7, 2026 A robots.txt file can control whether many automated crawlers are permitted to request specific URLs. It cannot guarantee AI visibility, and it should not be used as a generic switch for every form of AI access. Modern platforms increasingly separate search crawlers, model-training crawlers and user-triggered fetchers, which means a single blanket rule can create an outcome the organization did not intend. The safest approach is to treat robots.txt as a policy document implemented in code. First define what the organization wants for search visibility, model development, user-directed retrieval and private content. Then map those decisions to each platform's current documented user agents. Finally, validate that the CDN, WAF and application behave consistently with the file. **robots.txt is a crawl-control mechanism, not an AI visibility switch. Decide the policy first, then write the file.** ## What robots.txt Actually Controls Google defines robots.txt as a file that tells crawlers which URLs they may access on a site. It is primarily a crawl-management mechanism. Google explicitly warns against using robots.txt as the method for keeping a web page out of Google Search. If the objective is to prevent indexing, a noindex directive or authentication is usually the relevant control. That distinction becomes even more important for AI search. Blocking one crawler may stop automated discovery by that crawler, but it does not necessarily remove a URL from every search index, every third-party source, every user-triggered fetch or every AI product. A sound [professional SEO services](https://enginely.ai/services/seo/) programme therefore treats robots.txt, noindex, canonical tags, authentication, preview controls and firewall rules as different controls with different jobs. ## Start With a Four-Column Policy Matrix Before writing any directives, create a simple matrix for the platforms that matter. The rows are platforms or user agents. The columns are the business uses you need to decide. **1. Search Visibility** - **Question to answer:** Do you want this platform to discover and display your public pages in search results? - **Typical owner:** SEO or marketing team. **2. Model Training** - **Question to answer:** Should the platform be allowed to collect publicly available content for AI model development? - **Typical owner:** Legal, policy, or executive leadership. **3. User Retrieval** - **Question to answer:** Should the AI platform be permitted to fetch your public pages when a user specifically requests them? - **Typical owner:** Product, legal, or security team. **4. Infrastructure Access** - **Question to answer:** How should verified AI bots be handled by the web application firewall (WAF), CDN, rate limiting, and other infrastructure controls? - **Typical owner:** Security and engineering teams. This step prevents a common mistake: an SEO team blocks a training bot for a policy reason but accidentally blocks the separate search crawler, or a security team allows a named bot in robots.txt while the WAF still challenges every request. ## The Platform Decisions Are Not Identical ### OpenAI OpenAI says OAI-SearchBot and GPTBot can be controlled independently. OAI-SearchBot is used for ChatGPT search visibility. GPTBot relates to content that may be used to train OpenAI's generative AI foundation models. A site can therefore express a search-allow and training-disallow policy if that is the organization's decision. ### Perplexity Perplexity documents PerplexityBot for search discovery and Perplexity-User for user actions. Its current documentation says the user-triggered fetcher generally ignores robots.txt because the user requested the fetch. That means robots.txt is not the only access consideration for the product. ### Anthropic Anthropic separates ClaudeBot for potential model training, Claude-SearchBot for search quality and Claude-User for user-directed retrieval. Anthropic says its bots honour robots.txt and that disabling Claude-SearchBot may reduce visibility and accuracy in user search results. ### Google Google says Googlebot is the robots.txt control for Google Search, including AI Overviews and AI Mode. There is no separate special crawler you need to allow for those Search AI features. Google also says a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. ### Microsoft Bing Bingbot remains the standard crawler for Bing web search. Microsoft says it respects robots.txt and provides tools to verify Bingbot and test robots rules. Bing Webmaster Tools now also reports AI citation activity across supported Microsoft AI experiences, which helps separate crawl control from actual downstream citation. ## Four robots.txt Patterns and What They Mean The examples below are implementation patterns, not universal recommendations. The correct file depends on your policy, platforms and site architecture. Always verify the current user-agent names in official documentation before deployment. ### Pattern 1: Allow search crawlers and training crawlers User-agent: \* Allow: / Sitemap: https://www.example.com/sitemap.xml If there are no specific disallow rules for a crawler, crawl access is generally allowed. This is the simplest policy for a public site that does not want crawler-specific restrictions. It does not guarantee indexing or AI citations. ### Pattern 2: Allow OpenAI search, decline GPTBot training User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / User-agent: \* Allow: / This expresses two different OpenAI preferences. OAI-SearchBot is allowed for ChatGPT search. GPTBot is disallowed for potential model-training crawling. OpenAI documents these settings as independent. ### Pattern 3: Allow search bots while blocking selected training bots User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / This is an example of a search-visible, training-restricted posture across selected platforms. It is not complete for every crawler on the web and should not be copied without governance review. ### Pattern 4: Restrict one private path for all compliant crawlers User-agent: \* Disallow: /private-preview/ Allow: / Sitemap: https://www.example.com/sitemap.xml This can reduce automated crawling of a path, but it does not make the content private. A publicly reachable URL can still be shared, linked, cached elsewhere or requested by systems that do not follow the directive. Sensitive information belongs behind authentication, not behind robots.txt. ## Where robots.txt Must Live Google's documentation is explicit: robots.txt applies to the protocol, host and port where the file is published. A file at https://example.com/robots.txt does not automatically control https://shop.example.com/ or http://example.com/. 1. Audit every subdomain that hosts indexable or source-worthy content. 1. Check staging, documentation, help-centre, careers and regional hosts separately. 1. Do not assume the CMS robots setting covers assets served from another hostname. 1. Keep sitemap references current and use the preferred canonical host. Multi-location and multi-brand organizations are particularly vulnerable to mismatched policies because separate hosts may be managed by different teams. ## The WAF Can Override Your Good Intentions A robots.txt file is only one layer. A crawler can be explicitly allowed and still receive a 403 from the web application firewall, a JavaScript challenge from the CDN, a 429 from rate limiting or a 5xx response from overloaded infrastructure. This is why an AI crawler policy should include a verification and infrastructure rule. OpenAI and Perplexity publish IP ranges for relevant crawlers. Google and Bing publish verification guidance. Security teams can use those sources to distinguish genuine requests from spoofed user-agent strings. For organizations using [agentic SEO services](https://enginely.ai/services/agentic-seo/), crawler checks can be monitored repeatedly so a WAF or deployment change does not block visibility for weeks before a monthly audit notices it. ## robots.txt Is Not the Same as noindex **1. robots.txt** - **Primary purpose:** Controls how compliant crawlers access specific URL paths on your website. - **Important limitation:** It does not make public content private and is not Google's recommended method for preventing pages from being indexed. **2. noindex** - **Primary purpose:** Instructs supporting search engines not to include a crawled page in their search index. - **Important limitation:** The crawler typically needs to access the page first in order to see and apply the `noindex` directive. **3. Canonical** - **Primary purpose:** Indicates the preferred version of a page when duplicate or very similar URLs exist. - **Important limitation:** A canonical tag is a ranking signal, not an access control or privacy mechanism. **4. Authentication** - **Primary purpose:** Restricts access so that only authorized users can view protected content. - **Important limitation:** Requires appropriate product design, authentication workflows, and user access management. **5. Snippet Controls** - **Primary purpose:** Control how content can appear in Google Search snippets and previews. - **Important limitation:** They influence search result presentation only and do not prevent public access to the underlying webpage. ## Google AI Search Needs Googlebot Access, Not a Special AI File Google states that the same SEO foundations used for Search apply to AI Overviews and AI Mode. To be eligible as a supporting link, a page must be indexed and eligible to appear in Search with a snippet. Google says there are no additional technical requirements and no special AI text file or special schema required. For site owners, this removes a lot of unnecessary complexity. Do not weaken an otherwise healthy Search configuration to chase an invented Google AI bot. Preserve Googlebot access, keep important content indexable and use the standard preview controls when the objective is to limit what Search can display. This is consistent with Enginely's [generative engine optimization services](https://enginely.ai/services/generative-engine-optimization/): GEO adds cross-platform measurement and source work, but it does not replace the technical search foundation. ## A Safer Change-Control Process for robots.txt 1. Record the business reason for the change. 1. Name the exact crawler and official documentation used to identify it. 1. Check whether the crawler's search, training and user-fetch roles are separate. 1. Review all current user-agent groups so a new specific group does not accidentally omit existing rules. 1. Test priority URLs against the proposed file before deployment. 1. Deploy during a monitored window and preserve the previous version. 1. Verify the live file from outside the CMS or source-control environment. 1. Check logs, Search Console or Bing Webmaster Tools after the change. 1. Re-test the AI search prompt panel on the normal measurement cadence rather than immediately declaring success. The important governance principle is reversibility. robots.txt is easy to edit, but a mistake can affect discovery across thousands of URLs. Treat it like production configuration, not copywriting. ## Common AI robots.txt Mistakes ### Mistake 1: Blocking every bot with AI in the name This ignores the difference between search discovery and model training. The organization may unintentionally remove itself from a search surface it wanted to use. ### Mistake 2: Assuming Disallow makes content confidential robots.txt is public and voluntary. Sensitive information must be protected with access control. ### Mistake 3: Editing only the main domain The relevant practitioner, documentation or product pages may live on a subdomain with a different file. ### Mistake 4: Trusting the user-agent string Spoofed bot traffic can carry the same name. Use current provider verification methods before creating firewall exceptions. ### Mistake 5: Measuring crawl access as visibility An allowed crawler can still ignore, fail to index, fail to retrieve or fail to cite the page. Access is only the first stage. ### Mistake 6: Forgetting the user-triggered fetcher Some products use separate user agents when a person explicitly asks the product to visit a page. The platform may document different behaviour for these requests. ## How to Measure Whether a Policy Change Helped Do not judge the effect from one AI answer after deployment. robots.txt changes can take time to be fetched and processed, and generated responses vary. Use several evidence layers. - Technical evidence: the live file contains the intended rule. - Infrastructure evidence: verified crawlers receive the expected response codes. - Search evidence: priority pages remain indexed or discoverable where appropriate. - AI evidence: repeated prompt tests track mentions, citations and referenced pages. - Representation evidence: the organization is described accurately. - Commercial evidence: referral activity, branded demand and qualified enquiries are reviewed where measurable. Enginely's [SEO, GEO and AI search services](https://enginely.ai/services/) use this wider measurement chain so a robots.txt edit is never presented as an outcome by itself. ## Frequently Asked Questions ### Can I allow ChatGPT Search but block OpenAI training? OpenAI documents OAI-SearchBot and GPTBot as independent controls. A site can allow OAI-SearchBot for search while disallowing GPTBot for potential model-training crawling. ### Will robots.txt keep a page out of Google AI Overviews? Blocking Googlebot can prevent normal crawling, but Google does not recommend robots.txt as the method for keeping a page out of Search. Google says AI Overviews and AI Mode use Google Search eligibility and standard Search controls. ### Do I need an llms.txt file for Google AI visibility? Google says no special AI text file is required for AI Overviews or AI Mode. Standard technical SEO, index eligibility and useful content remain the documented requirements. ### Should I copy a competitor's robots.txt AI bot rules? No. Their legal position, platform priorities, CMS, subdomains and WAF configuration may differ from yours. Use the competitor file only as a clue to investigate, not as a policy template. ## Write robots.txt From Policy, Not Fear AI has made robots.txt more visible to marketing teams, but the file has not become magical. It remains a crawler-access control whose meaning depends on the user agent, the path and the platform's current implementation. The durable process is straightforward: define the organization's search and training choices, map those choices to official crawler documentation, implement the rules carefully, verify the infrastructure and measure downstream visibility separately. That approach is slower than copying a viral robots.txt block, but it is far less likely to remove a business from the very search experiences it is trying to understand.