How 9 of 9 Publishers Block AI Crawlers

We parsed 65 robots.txt files for 18 AI crawlers, and every publisher in the sample blocked them while not one software company did.

There is no middle ground in the data at all. If readers are the product, you block. If software is the product, you do not.

robots.txt used to be housekeeping — here are the folders I would rather you skipped. It is now where companies declare, in public, whether their work may be used to train and answer for a machine. We re-ran the whole survey from scratch on 6 September 2026. Here are the 6 lessons.

Check who blocks in your own market

We requested robots.txt from 67 domains, got 65, and parsed each one by user-agent group against 18 named AI crawlers.

About a quarter block at least one. And they are not spread evenly:

Every publisher we checked blocks, and most of them block almost the entire list. The New York Times and wikiHow stop 17 of the 18 crawlers we tested.

These are the same publishers who, in a companion survey we ran the same week, have not written an llms.txt file. They are not undecided about AI. They have decided the other way.

So before you touch your own file, go and read the files of the 5 companies you compete with. It takes ten minutes, and it will tell you more about what your market has concluded than any survey including this one.

Ask whether being found is your model

Now the other side, and it is absolute.

Developer tools, SEO tools and AI labs alike: not one site-wide block on a single AI crawler between them.

GitHub, GitLab, Stripe, Vercel, Cloudflare, Docker, Supabase, Sentry, Twilio. Ahrefs, Semrush, Moz, SE Ranking, Surfer, Screaming Frog, Sitebulb, and us. Anthropic, OpenAI, Perplexity, Mistral, Cohere, Hugging Face.

The logic is not complicated. A documentation page that a model reads and then recommends to a developer is doing its job. A news article that a model reads and then answers instead of you is not.

So your answer follows from your business model, not from a best practice. Work out which of those two things your pages are before you copy anybody's file.

If your pages exist to be found and acted on, blocking the crawler removes you from the answer and protects nothing. If your pages are the product, the calculation reverses completely — and no generic advice can make that call for you.

Never block the crawler that returns links

Among the blockers, Anthropic's crawlers are stopped slightly more often than OpenAI's — and at the bottom of the list sits OAI-SearchBot, the least-blocked crawler in the survey.

That is the one that powers search results with links back. It is the crawler most likely to send you a visitor, and it is the one fewest sites stop.

Which is the distinction to hold on to when you edit your own file. A crawler that trains a model and a crawler that returns a citation are not the same decision, even though they arrive from the same company and get pasted into the same list.

An important correction, because we got this wrong before. An earlier version of this article reported that every publisher blocking Anthropic was simultaneously allowing OpenAI, and tied it to licensing deals. Re-run against 65 domains, that pattern is much weaker — only 2 sites do exactly that. The tilt is real and consistent. The clean story was not.

Reread the file nobody has reread

Top of the blocklist is not GPTBot. It is CCBot, blocked by more sites than any crawler belonging to an AI company.

CCBot is Common Crawl — a non-profit that has been archiving the web since 2008, years before anyone worried about training data. Below it sit meta-externalagent, Bytespider, Applebot-Extended and PerplexityBot.

These files accumulate. Somebody added CCBot years ago for a reason nobody now remembers, someone else pasted in a list from a blog post, and the file grew. Very few read like a decision made once, on purpose, about the current set of crawlers.

Yours is probably the same. Open it and check the date of the last real change — if you cannot find anyone who remembers making it, you are enforcing somebody else's opinion from an era that has ended.

The same drift shows up in every technical file nobody rereads — it is exactly what we found auditing sitemaps in the same week, and what our Website Audit is checking when it looks at whether your robots rules and your indexable pages agree.

That is free and it covers the AI side. You also want to know what ordinary search sends you before you block anything, and Semrush and SE Ranking both show that clearly.

Parse the groups, do not grep the names

This is the methodological trap, and it will give you the opposite answer.

A robots.txt is organised into groups. Consecutive User-agent: lines share the rules that follow them, and a new group begins at the next User-agent: after a rule. A crawler is only blocked when a group naming it contains a site-wide Disallow: /.

Across our 65 files, 91 crawler mentions sit inside groups that do not block them — usually a large permissive group whose only rules are path-level, naming dozens of bots to grant them the same ordinary access as everyone else.

Grep for the bot name and you would count every one of those as a block. That is not a rounding error; on some sites it inverts the finding entirely.

So if you are auditing this at scale, parse the groups rather than searching for strings. Our survey script does, and it is in the repo if you want to run it against your own list.

And the same trap applies to your own file. A bot you believe you blocked may be named in a group that grants it everything — you will only find out by reading the group it sits in, not the line it appears on.

Read the file before you copy it

The instruction, and it is the whole practical takeaway.

Almost every robots.txt blocklist in circulation was pasted from another site. That is why CCBot outranks GPTBot, why 91 mentions are not blocks, and why so many files name crawlers that no longer exist.

Open yours. For each AI crawler in it, answer one question: would a visitor arriving from this system be worth having? For a subscription publisher the answer is often no. For anything sold by being found, it is almost always yes — and OAI-SearchBot, the one that returns links, is the one you least want to stop.

Then check what you are already getting from those systems before you change anything. Our AI Visibility Checker asks 6 buyer questions across the major assistants and counts mentions and citations separately, which tells you whether there is traffic to protect in the first place.

What we could not measure

And here is what this survey does not show. A couple of the tools named on this site are partners of ours — if you buy through those links we earn a commission, and you do not pay a cent more.

robots.txt is a request, not a control. It records what a site asks for. It does not tell you whether any crawler obeyed, and we have no way to verify compliance from outside. A blocked crawler in this data is a stated intention.

We tested 18 named crawlers and counted a block only as a site-wide Disallow: / inside a group naming that agent. Path-level rules, crawl-delay and wildcard agents are not counted, so a site that blocks an AI crawler from most of its site but not all of it reads here as not blocking.

2 of the 67 domains did not return a file at all — npmjs.com and rust-lang.org — so the denominators are 65.

The sample is 67 domains we chose across 6 categories, weighted towards companies an SEO audience would recognise, and it is not a random sample of the web. A different list gives a different rate. The method is scripts/survey-ai-crawler-files.mjs in our repo and the raw output is in evidence/surveys/.

We have no visibility into private licensing agreements, and this version of the article makes no claim about them. The earlier version did, on the strength of a much smaller sample, and that claim has been removed rather than restated.

Frequently asked questions

The one number to take away

31 to 0.

That is how many software companies in this survey block an AI crawler: none of them, against 9 out of 9 publishers.

There is no consensus best practice hiding in this data, because the two groups are not answering the same question. A publisher is asking whether a machine may substitute for its product. A software company is asking whether a machine may recommend its product.

Nobody can answer that for you, and no blocklist you paste from a blog post has answered it either. The most-blocked crawler in this survey belongs to a non-profit archive that predates the entire debate, which tells you how much of what is in these files was decided rather than inherited.

Open your robots.txt. It takes 5 minutes, and it is currently making a statement on your behalf that you may never have made.

Measured 6 September 2026. We requested robots.txt from 67 domains, parsed 65 of them by user-agent group, and counted a crawler as blocked only when a group naming it contained a site-wide Disallow: /. The 18 crawlers tested were GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-User, Google-Extended, CCBot, PerplexityBot, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot, cohere-ai, Diffbot, omgili, ImagesiftBot and Timpibot. The method is scripts/survey-ai-crawler-files.mjs in our repo and the raw results are in evidence/surveys/, so every figure here is re-runnable.