robots.txt and AI Crawlers: How to Control Access to Your Site

Over the past two years new names have appeared in server logs: GPTBot, ClaudeBot, PerplexityBot. These are language-model crawlers, and they read pages much as search robots do. The question every site owner ends up asking: let them in or not.

The answer depends on how you make money. But first it helps to know who these crawlers are and how they differ, because they are easy to confuse and the cost of a mistake varies.

Who is reading your site

The main confusion is that one company may run several crawlers with different purposes.

The difference between the first two rows matters most. Blocking GPTBot keeps you out of the training set. Blocking OAI-SearchBot removes you from ChatGPT answers that cite sources. People often want the former and end up doing both.

Google-Extended is not a crawler

This one deserves an explanation. Google-Extended does not visit sites: it is a switch in robots.txt declaring whether your content may be used to train Gemini.

Importantly, it has no effect on ordinary search. Googlebot handles indexing, so you can block Google-Extended and stay in the results.

Should you allow access

There is no universal answer here, and anyone claiming otherwise is simplifying.

For: if you sell services people look for in chat, being cited with a link brings customers. For expert content this is a new channel few use deliberately yet.

Against: if content is your product, handing it over for training means working for someone else's model. For licensed material or third-party rights it becomes a legal question too.

What you certainly should not do is decide blindly. Check your current AI crawler access and see what is already configured. Often it turns out somebody blocked half of them once and forgot.

What robots.txt does not do

Worth stating plainly, because the topic attracts overstatement.

It does not affect Google rankings. Allowing AI crawlers does not lift your site in search. These are separate systems, and claims to the contrary have no basis.

It is not a technical block. robots.txt is a convention. Major companies honour it and publish their crawler names. Smaller scrapers often do not. Real protection needs server-level rules.

It does not cover subdomains. Every subdomain needs its own file. Rules on the main domain do not reach shop.example.com.

How to configure it

Each crawler needs its own group — the wildcard rule applies only when a crawler has no group of its own:

Names must match exactly what the vendors publish. GPT-Bot instead of GPTBot and the rule does nothing — and without a check that is hard to spot.

Verifying the change

1. Run the site through the AI crawler checker, which shows the verdict for each one separately.
2. Confirm the search-engine rules survived: the robots.txt tester shows which rule applied to Googlebot.
3. Check the file returns a 200. A server error instead of the file means there are no rules at all.

Google re-reads robots.txt with a delay, usually within a day. Other crawlers may cache it longer.

In short

Work out which crawlers you actually want, since they differ even within one company. Set up separate groups, verify the result, and do not expect it to raise your Google rankings. This is about appearing in model answers, not about search.