A robots.txt file can ask AI training crawlers to leave your photographs alone while search engines carry on indexing them. It is a request, not a barrier, so know exactly what you are asking.
A robots.txt file is a short text document at the root of a website that tells automated visitors which parts of the site they may fetch. For an image owner it can do one useful thing in 2026: ask the crawlers that gather AI training data to stay out, while leaving search engines free to index the same pictures. It does this by request only. Nothing in the file can refuse a connection.
This article is general information, not legal advice. Checked on 18 November 2026.
What is robots.txt, and is it binding?
The convention dates from the 1990s and was written up as an internet standard, RFC 9309, in September 2022. That document is frank about its limits. The rules "are not a form of access authorization", and the protocol "is not a substitute for valid content security measures". A crawler reads the file, finds the group addressed to its own name and decides whether to comply. Names are matched without regard to capital letters, paths are case sensitive, and a crawler may cache the file for up to 24 hours, so changes are not instant.
For anything that must not be collected, the answer is access control: a login, a server rule, or not putting the file online. robots.txt is for stating preferences to bots that are willing to listen.
Which AI crawlers should image owners know?
The useful distinction is between crawlers that gather training data, crawlers that build a search index, and fetchers that act for a user in the moment. The table gives the names the companies themselves publish.
| Token | Operator | Stated purpose | Notes |
|---|---|---|---|
| GPTBot | OpenAI | Content that may be used to train generative models | Disallowing it "indicates a site's content should not be used in training" |
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search | Separate from training |
| ChatGPT-User | OpenAI | Fetches made when a user asks | OpenAI says "robots.txt rules may not apply" |
| ClaudeBot | Anthropic | Content that could contribute to model training | Anthropic says it supports the non-standard Crawl-delay line |
| Claude-SearchBot, Claude-User | Anthropic | Search quality; user-requested fetches | Blocking may reduce visibility in Claude's answers |
| Google-Extended | Controls use for training Gemini models and for grounding | A token, not a crawler; no effect on Google Search | |
| Applebot-Extended | Apple | Controls use of Applebot data for Apple's foundation models | Apple says it "does not crawl webpages" |
| meta-externalagent | Meta | Purposes including training AI models | Meta's user-request fetcher "may bypass robots.txt rules" |
| CCBot | Common Crawl | An open repository of web crawl data, widely reused | Opt out with a CCBot group |
The table records what each operator says its token does. It is not evidence that any crawler complies in practice, and we have not tested them.
Other operators publish tokens too, among them PerplexityBot, Bytespider and Amazonbot. Check each operator's own documentation before relying on a name, because lists copied from blog posts go stale quickly.
How do I block AI training but keep Google Images?
Put training tokens in one group and leave search crawlers alone. Several user-agent lines can share a single set of rules:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /images/
User-agent: *
Allow: /
Three cautions. A crawler obeys the most specific group that names it and ignores the general one, so any general rules you need, such as keeping bots out of an admin folder, must be repeated inside the named group. The file applies per hostname, so a separate image subdomain needs its own copy. And blocking Googlebot-Image, as opposed to Google-Extended, removes pictures from Google Images altogether, which is seldom what a photographer selling licences wants. If search visibility matters to you, the companion piece on the Licensable badge in Google Images covers the other half of the job.
Whether to refuse the AI search and user agents as well is a commercial judgement. Refusing them keeps your pages out of AI answers; allowing them may send readers your way. The training tokens are the ones that concern reuse of the photographs themselves.
What robots.txt cannot do
- It cannot compel. A crawler that ignores the file meets no technical obstacle. On 4 August 2025 Cloudflare alleged that Perplexity had used undeclared crawlers that disguised their identity to reach sites that had disallowed its declared bot; Cloudflare removed it from its verified bot list. That is one company's published allegation, not a court finding.
- It cannot follow the image. The file speaks for your site. A photograph reproduced by a client, a news site or a pirate is covered by their file, not yours.
- It cannot reach the past. Datasets assembled before you added the rule are unaffected.
- It works by site and path, not by picture. You cannot mark one photograph inside a page for different treatment except by putting it in a different folder.
- It cannot be seen in the file. Once a picture is downloaded, nothing in it records that your site asked for it not to be used.
The last two gaps are why embedded metadata matters. The IPTC Data Mining property puts the reservation inside the image, where it stays until someone strips it, and the free AI training opt-out metadata tool writes it without uploading the file. The two signals fail in different circumstances, which is the argument for using both. The wider picture is in how to protect your photographs from AI scraping.
Does robots.txt have legal weight?
Decided. In the EU, Article 4(3) of Directive 2019/790 lets rights holders reserve text and data mining rights by machine-readable means, and the general-purpose AI Code of Practice, published on 10 July 2025, commits its signatories to honour robots.txt. It is the only protocol the Code names. In the UK there is no opt-out regime for robots.txt to plug into: the proposal was dropped in the government's report of 18 March 2026, which also noted that robots.txt works at site level and not at the level of the individual work. In the US we know of no statute that gives the file a defined effect.
Pending. The European Commission consulted between December 2025 and January 2026 on which other machine-readable protocols AI providers should respect, and a final list could not be confirmed on the date above. Newer proposals sit alongside robots.txt without replacing it: RSL 1.0, published on 10 December 2025, adds machine-readable licence terms, and an IETF working group has been drafting a vocabulary for AI preferences that had not become a published standard when last checked. The article on text and data mining exceptions in the UK, EU and US sets out the surrounding law.
A robots.txt rule is therefore evidence of a stated position everywhere, and in the EU it may also be a legally relevant reservation. It is not a contract and should not be described as one.
A short checklist
- Confirm the file exists at the root of every hostname that serves images.
- Name the training tokens explicitly. Do not rely on the wildcard group.
- Repeat any general disallow rules inside each named group.
- Leave Googlebot and Googlebot-Image alone unless you intend to leave search.
- Add a comment line with a licensing contact. Humans read these files too.
- Check your server logs to see which bots visit and whether they comply.
- Pair the file with embedded rights metadata, and verify a sample in the metadata viewer.
- Review the list every six months.
How Archive.London uses robots.txt
Archive.London's own file follows the pattern above. It allows search and user-requested agents across the public site, and asks the training crawlers it names not to collect the photographs. It opens with a comment stating that the pictures are not licensed for text and data mining or machine learning, and pointing to the licensing page. This is a signal, and the site treats it as one: only a small, watermarked selection from collections such as music and jewellery is published, while more than half of the archive is still on film and has yet to be scanned.
The Robert Harper Photograph Library is owned by 825 Ventures Ltd, the owner and rights holder. Licences for AI and data uses are negotiated directly with the single rights holder, subject to rights review, and teams who want to ask instead of crawl will find the route on the AI training data licensing page.
Frequently asked questions
How do I block GPTBot from my images?
Add a group to the robots.txt file at the root of your site: a line reading 'User-agent: GPTBot' followed by 'Disallow: /' to refuse the whole site, or 'Disallow: /images/' to refuse one folder. OpenAI says disallowing GPTBot indicates that a site's content should not be used in training. The file must be repeated on each subdomain you want covered.
Will blocking AI crawlers hurt my Google ranking?
Not according to Google, if you block the right tokens. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. Blocking Googlebot or Googlebot-Image is a different matter and would remove pages or pictures from search. Keep training tokens and search crawlers in separate groups so one rule does not catch the other.
Is robots.txt legally binding on AI companies?
Not as such. It is a technical convention. In the EU, signatories to the general-purpose AI Code of Practice commit to honouring robots.txt as a machine-readable rights reservation. In the UK and US no statute gives the file a defined effect. Its legal weight is still being worked out. This is general information, not legal advice.
What is the difference between GPTBot and ChatGPT-User?
GPTBot crawls automatically and gathers content that may be used to train OpenAI's models. ChatGPT-User fetches a page only when a person using ChatGPT asks for it. OpenAI says ChatGPT-User is not used for automatic crawling and that robots.txt rules may not apply to it, because the request is started by a user.
Does robots.txt remove my photos from AI models that already trained on them?
No. The file affects future crawling by bots that obey it. It does not reach back into datasets collected earlier, and it does not touch copies of your photographs held on other websites. Anthropic, for example, describes a block on ClaudeBot as a signal about a site's future materials.
Sources
- RFC 9309: Robots Exclusion Protocol (September 2022)
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google Search Central: Google's common crawlers
- Apple: About Applebot
- Meta for Developers: Meta web crawlers
- Common Crawl: CCBot
- Cloudflare blog: Perplexity is using stealth, undeclared crawlers (4 August 2025)
- European Commission: General-Purpose AI Code of Practice
- Directive (EU) 2019/790 on copyright in the Digital Single Market
- Reed Smith: UK copyright and AI report, the opt-out is dead
- RSL 1.0 specification announcement (10 December 2025)
Published by Archive.London. Corrections are welcome under our editorial policy.

