AI Search Bots: A Practical robots.txt Guide

For public pages you want customers to discover, check OAI-SearchBot, PerplexityBot and Claude-SearchBot. These are search crawlers. Training crawlers and tools that fetch a page at a user's request have separate roles and may have separate controls.

A correct robots.txt file helps express your preferences. It cannot guarantee that an AI service will visit, index or recommend your company, and it cannot protect a private system. This guide explains the distinctions, gives practical examples and shows how to verify what happens after a change.

Which bot does what?

OpenAI: OAI-SearchBot supports ChatGPT search. GPTBot collects content that may be used for foundation-model training. ChatGPT-User handles certain requests initiated by users; OpenAI says robots.txt rules may not apply to those requests. OAI-SearchBot and GPTBot preferences are independent. OpenAI crawler documentation.

Perplexity: PerplexityBot surfaces websites in search results and is not used to collect content for foundation-model training. Perplexity-User is a separate fetcher for user actions. Consult the operator's current documentation for its behavior instead of treating every request with “Perplexity” in the name as the same crawler. Perplexity crawler documentation.

Anthropic: Claude-SearchBot supports search quality; ClaudeBot collects material that could contribute to training; Claude-User retrieves content in response to user requests. Anthropic says its bots honor robots.txt directives. Anthropic crawler documentation.

These are the operators' published purposes. A user-agent string in your logs is a claim of identity, so verify the source using the operator's current guidance before making an access decision.

How do you allow the three search crawlers?

Serve a plain-text file at /robots.txt on the relevant host. For a public site that intends to allow these search crawlers across all its pages, the explicit groups are:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

This is an example to adapt to your existing policy, not a complete replacement for an established robots.txt file. Preserve any sitemap reference and deliberate restrictions. Check each subdomain separately; a rule on your marketing site does not configure a separate app or client portal.

In our September 7, 2026 public audit, Rebel's robots.txt already explicitly allowed all three search bots. That meant adding more Allow lines was not the next useful improvement. Article discovery and accurate content were more practical priorities.

How do you allow search while restricting training?

The following example expresses separate preferences for OpenAI's two crawlers:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

OpenAI documents this separation. Decide the training preference deliberately; a desire for organic enquiries does not require automatically changing it. Blocking a crawler also does not erase information previously collected or guarantee removal from every answer.

For Anthropic, the corresponding tokens are Claude-SearchBot and ClaudeBot. Review each operator separately before applying a common “block AI bots” switch at your host, because the switch may affect the search access you intended to preserve.

How does rule matching actually work?

Robots.txt does not operate as a simple list where the first matching line wins. Under the Robots Exclusion Protocol, matching bot groups are combined; the wildcard group is used when no specific group matches. For path rules, the most specific match takes precedence, and equivalent Allow and Disallow rules should favor Allow.

Google documents its own matching behavior explicitly: it selects the most specific user-agent group, combines equally applicable groups, and does not merge that group with the wildcard group. The order of groups does not establish priority. Google's robots.txt specification.

Consider this example for a bot that follows those group rules:

User-agent: *
Disallow: /drafts/

User-agent: OAI-SearchBot
Allow: /

Do not assume the /drafts/ restriction is inherited by OAI-SearchBot. Put any intended restriction in its applicable group as well.

For a path exception within one group:

User-agent: OAI-SearchBot
Disallow: /resources/
Allow: /resources/public/

The longer /resources/public/ rule permits that subsection for a compliant parser. This works because the matching path is more specific, not because Allow always outranks Disallow. Test actual URLs, including capitalization, against the implementation you use.

What robots.txt cannot do

Protect private data: robots.txt is public and is not an authorization mechanism. Use authentication and appropriate permissions for client dashboards, internal tools and unpublished operational data. Listing a path in robots.txt can reveal that it exists. RFC 9309 security considerations.

Guarantee removal from search: crawling and indexing are different. For Google, a crawler must be able to fetch a page to see its noindex directive. Do not assume a robots block plus an unread noindex tag provides a redundant removal method. Other operators have their own supported controls. Google's page-level control guidance.

Replace other technical checks: an allowed bot may still receive a firewall challenge, a 403, a broken redirect or empty content. Conversely, a successful test request does not prove that the real crawler can reach your pages from its own network.

Turn llms.txt into a crawl policy: that file is an optional content overview for systems that use it. It does not replace robots.txt or access controls. Our structured-data guide explains the different purposes.

How should you verify a change?

1. Save a baseline. Record the current file, intended allowed and restricted paths, and the reason for each rule. Keep a dated copy of the previous configuration.
2. Inspect the deployed response. Check that /robots.txt returns the intended text on the correct host. Confirm that a deployment or host setting has not replaced it.
3. Test the rules against real examples. Include the homepage, a service page, an article and any restricted public path. Check specific bot groups as well as the wildcard group.
4. Inspect hosting logs. Record request time, path, response status and available crawler-verification evidence. Separate successful content requests from blocked requests and robots.txt fetches. An allowed bot is not obliged to visit, so no activity does not establish a fault.
5. Measure visibility separately. Once access is sound, check actual citations and qualified visits using a repeatable query set. See our AI visibility testing guide.

Do not promise a universal recrawl interval. OpenAI says robots.txt changes can take about 24 hours to affect its search systems, and Perplexity documents up to 24 hours for its settings. Those are policy-update timings, not promises to crawl or index every changed page. Use observed logs and current operator documentation to investigate a specific case.

Start with Rebel's free AI Search Readiness Audit to inspect a public page's technical signals. If your concern is separating a public website from an internal workflow or client tool, our AI Operations service starts with the system, access requirements and workflow you need to support.