GPTBot vs OAI-SearchBot vs ChatGPT-User: correct robots.txt setup

If you want to be discoverable in ChatGPT search, allow OAI-SearchBot. Allowing GPTBot is not necessary for this purpose, since it relates to the possible use of content for model training. ChatGPT-User represents a request triggered by a user and may not behave like a regular automated crawler.

Overview of OpenAI's bots

User-agent	What it's for	What happens if blocked
OAI-SearchBot	ChatGPT search	Content won't be used in ChatGPT Search summaries and citations; only a navigational link may appear
GPTBot	Possible improvement and training of foundation models	Content from this crawl won't be used for that training purpose
ChatGPT-User	Page visit requested by a user	The user-triggered fetch may fail; this is not an automatic crawl for indexing

OpenAI states that these rules are independent of one another. So you can allow Search while disallowing training.

Recommended configuration: Search yes, training no

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This setup suits a website that wants visibility in ChatGPT's current answers but doesn't want to let the GPTBot crawler use its content for training.

Allow both crawlers

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

An explicit Allow: / isn't always necessary if there's no broader blocking rule in place. Still, an explicit rule is clearer during an audit.

Allow only a selected part of the site

User-agent: OAI-SearchBot
Allow: /znalostni-baze/
Allow: /sluzby/
Disallow: /ucet/
Disallow: /interni-vyhledavani/
Disallow: /kosik/

Robots.txt is not a security tool. Protect sensitive content with authentication. Disallow only tells well-behaved bots not to crawl a URL.

When to use noindex

Robots.txt controls crawling, while noindex asks the system not to include a page in its index. For a bot to read the meta tag, it must first be able to visit the page.

<meta name="robots" content="noindex, follow">

OpenAI notes that a blocked page may in some situations still appear as a bare navigational link if its URL is known from another source. To prevent that from happening, use noindex while also letting the crawler fetch the page.

Robots.txt isn't enough: check your CDN and firewall

A common problem looks like this:

• robots.txt allows the crawler;
• a regular browser opens the page just fine;
• but a security layer returns a 403, 429, or a CAPTCHA to the bot.

Check:

1. server and CDN logs;
2. rules blocking unknown bots;
3. rate limiting;
4. geo-blocking;
5. anti-automation protection;
6. OpenAI's published IP ranges.

A user-agent string can be spoofed. To allow access at the firewall level, use the official IP lists and a properly configured security policy, not just the header text.

How to verify robots.txt

Open:

https://yourdomain.com/robots.txt

Then look for:

• a blanket User-agent: * rule combined with Disallow: /;
• a dedicated block for OAI-SearchBot;
• conflicting rules;
• a blocked knowledge base section;
• blocked CSS, JS, or API resources needed to render the content.

Example of a problematic configuration:

User-agent: *
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Different parsers may apply the rules according to the specification and the most specific matching block. Even so, it's better to keep the configuration simple and then verify it with an actual request and in the logs.

What to monitor in server logs

For every request, log:

• timestamp;
• requested URL;
• user-agent;
• IP address;
• status code;
• response size;
• response time.

Watch especially for:

• repeated 403 or 429 responses;
• redirect chains;
• 5xx errors;
• requests for robots.txt alone with no subsequent crawl;
• crawling of parameterized or infinite URLs;
• content that differs between bots and users.

When the change takes effect

OpenAI states it takes roughly 24 hours for its systems to adapt to a robots.txt change. That doesn't mean a page will be cited within 24 hours. It only means the new allow or disallow rule may be reflected in the crawler's behavior by then.

Common mistakes

• Using GPTBot instead of OAI-SearchBot in guidance meant for ChatGPT Search.
• Allowing access in robots.txt but blocking it at the CDN level.
• Blocking a page while also expecting the crawler to read its noindex tag.
• Publicly accessible sensitive URLs protected only by Disallow.
• Assuming that allowing a crawler guarantees a citation.
• Unbounded parameterized URLs creating a crawler trap.

Recommended audit process

1. Download and archive the current robots.txt.
2. Check the rules for all three OpenAI user-agents.
3. Run an HTTP test on key URLs.
4. Verify meta robots and the X-Robots-Tag header.
5. Check canonical tags and redirects.
6. Verify the firewall and CDN configuration.
7. Monitor logs for at least several days after the change.
8. Separately track the site's presence in ChatGPT Search.

Sources

• OpenAI: Overview of OpenAI crawlers
• OpenAI: Publishers and Developers FAQ