OpenAI Will Honour a Shared robots.txt Signal for AI Training Preferences

OpenAI has said it will honour a shared signal in robots.txt, the file that tells automated visitors what a website allows. The signal lets a website owner state whether content may be used to train generative AI models. OpenAI announced the plan on 30 October 2026 and calls it a voluntary effort to cut the work of keeping separate rules for each AI provider.[1]

OpenAI says it aims to implement the signal in the coming months and will publish details then. The signal will not replace its existing crawler controls, which continue to apply. The page also encourages other AI providers to honour the signal. It does not say which products or models the signal will cover.[1]

The signal comes from an Internet-Draft titled Machine-Readable Training Preference Signal. An owner writes an ordinary robots.txt group whose User-Agent line is the token __usage_ai-training. A Disallow rule then states a preference against AI training for those paths, and an Allow rule states a preference in favour. The draft's own example disallows a whole site except a press-kit folder.[2]

The draft adds no new directive or grammar. Parsers that do not know the token ignore the group, so existing crawler rules behave as before. A wildcard group, written User-Agent: *, must not be read as matching the token, so a blanket rule for all crawlers does not state this preference. If no group names the token, no preference has been expressed.[2]

The draft is an individual submission by K. Kelley, whose affiliation is listed as Anthropic. It is dated 30 October 2026, expires on 3 May 2027 and states an intended standards-track status. The datatracker page says the draft is not endorsed by the IETF and has no formal standing in the standards process.[2]

OpenAI says the signal draws on the IETF AI Preferences working group, whose vocabulary the draft follows for its definition of AI training. The draft says the token is not a directive for crawling. A consumer may use it to inform crawling, and private agreements may override a stated preference.[1][2]

OpenAI says its existing crawler controls remain in force, and its page links to documentation for them. Rules that name OpenAI's crawlers keep working. The new signal is meant to spare owners from naming each provider's crawler and maintaining separate rules for every one.[1]

The announcement does not say how OpenAI will treat content collected before the signal takes effect. It does not say which other companies have agreed to honour the signal, beyond a reference to other working group participants. Owners cannot yet assume that one line in robots.txt will reach every provider.[1]