More unique IPs than any network on the planet, in the world's largest residential pool. That's Shifter.More unique IPs than any network, in the world's largest pool. That's Shifter.

See the numbers

Who Has an llms.txt? We Checked the Top 10,000 Websites

We checked the top 10,000 websites for llms.txt in November 2026. 9.3% have a real one, adoption is rising fast, and those sites rarely block AI crawlers.

SEO & Search · Updated November 16, 2026 · 7 min read
top 10,000 websites9.3%have a real llms.txt5.9%in May 2026
One in eleven of the top 10,000 sites has a real llms.txt. Each amber square is about 1%.
Knowledge
On this page
  1. The headline numbers
  2. Half of the “200 OK” responses are not llms.txt files
  3. Bigger sites, and developer sites, lead
  4. The surprise: sites with llms.txt rarely block AI crawlers
  5. Does anyone read it?
  6. What to do with this
  7. How we did it
  8. The bottom line
  9. Sources and references

llms.txt is the most talked-about file in AI search, and one of the least measured. It is a Markdown file at the root of a site that tells AI tools what the site is and where its most useful pages are. Plenty of people recommend it; Google says you can ignore it. For teams that monitor search visibility, the useful question is simpler: who actually has one?

On 16 November 2026 we checked the top 10,000 domains in the Tranco ranking, the same list we used for our study of who blocks AI crawlers. Here is what we found.

The headline numbers

DomainsShare
Top 10,000 domains checked10,000
Running a website (answered our request)7,865
With a real llms.txt9299.3% of all, 11.8% of websites
Also with an llms-full.txt15016% of those with llms.txt

A vendor scan of the same kind of list in May 2026 found 586 valid files, 5.86% of the top 10,000. Methods differ slightly, but the direction is clear: adoption rose by roughly half in six months.

Half of the “200 OK” responses are not llms.txt files

2,068 domains answered our request for /llms.txt with a 200 OK status. Fewer than half of them, 929, served an actual llms.txt. The rest returned their home page, an error page dressed as a success, or a redirect to somewhere else. We also found 128 plain text files with no title, which the specification requires.

So counting status codes overstates adoption by more than double. We counted a file as real only if it was text, not HTML, stayed at an llms.txt address after redirects, and began with the Markdown title the specification calls “the only required section”.

Bigger sites, and developer sites, lead

RankWebsitesWith llms.txt
Top 1007312.3%
Top 1,00073315.8%
1,001 to 5,0003,04812.4%
5,001 to 10,0004,08410.7%

Among the earliest adopters in the ranking are Cloudflare, GitHub, Adobe, Samsung, Dropbox, PayPal, Shopify, Stripe, Cisco and Slack. The pattern is developer platforms and software companies: sites whose documentation people already paste into AI coding tools.

Domain endings tell the same story. 25.7% of .io websites have an llms.txt, and 15.4% of .com sites, against 2.5% of .edu, 1.8% of .jp and 1.0% of .gov. Docs platforms explain part of it: Mintlify has generated llms.txt for the docs sites it hosts since late 2024, and Yoast SEO added it for WordPress in June 2025.

The surprise: sites with llms.txt rarely block AI crawlers

We joined the results with the robots.txt files from our October study. The two groups look very different.

AI crawler blocked in robots.txtSites with llms.txtSites without
GPTBot (OpenAI)4.9%16.9%
ClaudeBot (Anthropic)4.2%15.7%
Google-Extended3.0%13.8%
CCBot (Common Crawl)6.5%18.0%
At least one training crawler7.4%20.7%

Sites that publish an llms.txt are about three times less likely to block AI training crawlers. That makes sense: an llms.txt is an invitation, and the companies sending it mostly want AI tools to understand their products.

A few sites send mixed signals. Twelve publish an llms.txt while blocking GPTBot, ClaudeBot and PerplexityBot in robots.txt, among them Bloomberg and the Daily Mail. The robots.txt data is from early October, so some may have changed since, but the point stands: an llms.txt lists content, it does not grant permission. Our guide to robots.txt and AI opt-outs covers the files that do.

Does anyone read it?

Here the picture is weaker than the adoption curve suggests.

  • Google Search does not use it. Google’s guide to AI features says you “can ignore tactics like ‘chunking’ content, creating unnecessary AI text files (like llms.txt)”, while noting that it is “completely fine” to create them for other services that use them. Earlier, John Mueller compared llms.txt to the keywords meta tag.
  • Chrome checks for it. Lighthouse now has an llms.txt audit, describing it as “an emerging convention used to provide a machine-readable summary of a website’s content”. A missing file is marked not applicable, “as providing the file is optional at the moment”.
  • Crawlers rarely fetch it. No major AI company documents its crawlers reading other sites’ llms.txt files. In Ahrefs’ log study, 97% of about 38,000 files received no requests at all in May 2026.

The main readers today are AI coding assistants and agents pointed at documentation, where a clean list of pages is genuinely useful.

What to do with this

  1. Add one if you have documentation or a product with many pages. It is cheap, and agents and coding tools do use it. Keep it short: a title, a one-line summary, and sections linking to your most useful pages.
  2. Do not expect rankings. Nothing suggests llms.txt affects Google Search or AI Overviews. Our guide to AI Mode and rank tracking covers what does move AI visibility, and residential proxies let you check how AI answers describe you in each market.
  3. Make it real. Serve plain text at /llms.txt, not your home page, and check after every CMS update. More than half the “200 OK” responses we saw were not llms.txt files.
  4. Keep the signals consistent. If your llms.txt invites AI tools to read your pages, check that robots.txt does not shut out the same tools, or the other way round.
  5. Measure, then decide. Watch your server logs for requests to /llms.txt. If nobody fetches it after a few months, you have your answer.

How we did it

On 16 November 2026 we requested https://<domain>/llms.txt, then the www. address if the first failed, for each of the top 10,000 domains in the Tranco ranking (list Q2K34). We used one request per address with a descriptive user agent, followed redirects, and read at most 2 MB. A domain counted as running a website if it returned any HTTP response. A file counted as a real llms.txt if it was not HTML, stayed at an llms.txt address and began with a Markdown title. The robots.txt comparison uses our scan of the same domains from 6 October 2026.

The bottom line

About one in eleven of the top 10,000 websites now has a real llms.txt, up roughly half since May, led by developer platforms and the biggest sites. Those sites are three times less likely to block AI crawlers. But Google Search ignores the file and crawlers rarely fetch it, so treat llms.txt as a cheap convenience for AI tools and agents, not as an SEO lever.

Sources and references

Frequently asked questions

How many websites have an llms.txt file?

In our November 2026 check of the top 10,000 domains in the Tranco ranking, 929 had a real llms.txt: 9.3% of all domains and 11.8% of those that run a website. Among the top 1,000, the share was 15.8%.

Does Google use llms.txt?

Not for Search. Google's guide to AI features says you can ignore tactics like creating unnecessary AI text files such as llms.txt, while adding that it is completely fine to create them for other services that use them. Chrome's Lighthouse does include an llms.txt audit, marked not applicable when the file is missing.

Do AI crawlers read llms.txt?

Rarely, so far. No major AI company documents its crawlers reading other sites' llms.txt files, and an Ahrefs log study found that 97% of about 38,000 llms.txt files received no requests at all in May 2026.

Should my site have an llms.txt?

It costs little and can help AI agents and coding tools find your key pages, which is why documentation sites lead adoption. Do not expect it to affect rankings in Google, and do not use it instead of robots.txt: it lists content, it does not grant or refuse permission.

Ready to get started?

Try Shifter's residential proxies, 205M+ IPs, 195+ countries, from $0.10/GB.

Get Started