The Internet Engineering Task Force (IETF) is developing new web standards to give site owners more control over how AI models use their content. The AI Preferences Working Group within IETF is creating machine-readable rules that allow site owners to specify how AI systems can access and use their content.
The group aims to standardize methods for expressing AI-related preferences, regardless of how those preferences are associated with content. They are also working on standard ways to attach these preferences to content using protocols like the Robots Exclusion Protocol and HTTP response header fields, as well as methods for resolving conflicting preferences.
Key takeaways
- The IETF's AI Preferences Working Group is developing machine-readable standards letting site owners state how AI systems may access and use their content.
- Two documents released in August set out the proposals: 'A Vocabulary For Expressing AI Usage Preferences' and 'Associating AI Usage Preferences with Content in HTTP'.
- The scheme categorises AI systems as 'search', 'train-ai', 'train-genai' and 'bots', each of which can be permitted or refused.
- Rules go in a new 'Content-Usage' field in robots.txt and can apply per folder — 'Content-Usage: train-ai=n' blocks all model training, while 'Content-Usage: /ai-ok/ train-ai=y' permits it for one subfolder.
- Existing efforts such as llms.txt have no confirmed adoption, and Google has said it does not support llms.txt.
Two documents released in August outline potential standards:
- A Vocabulary For Expressing AI Usage Preferences
- Associating AI Usage Preferences with Content in HTTP
These documents propose updates to the Robots Exclusion Protocol, introducing new rules that enable site owners to define how AI systems can use their content. The proposed system categorizes AI systems with labels such as “search,” “train-ai,” “train-genai,” and “bots,” and allows site owners to permit or disallow each category. These rules can be set at the folder level and customized for different bots, using a new “Content-Usage” field in robots.txt.
For example:
Content-Usage: train-ai=n means no content on the domain can be used for training any large language model.
Content-Usage: /ai-ok/ train-ai=y means content in the /ai-ok/ subfolder can be used for training models.
While initiatives like llms.txt exist, there is no confirmation that AI companies follow these rules. Google has stated it does not support llms.txt. With Google’s Gary Illyes involved in the IETF working group, there is hope that major tech companies will adopt the finalized standards.
Frequently asked questions
How is this different from the current robots.txt?
Today's robots.txt controls whether a crawler may fetch a page, not what it may do with it afterwards. These proposals separate the two, so a site can allow search indexing while refusing to have the same content used for model training.
Would AI companies actually follow it?
That is the open question, since compliance is voluntary. The encouraging sign is that Google's Gary Illyes is involved in the working group, which raises the odds that major companies honour the finalised standard.
Why does a standard matter rather than each site inventing its own rules?
Machine-readable preferences only work if crawlers agree on how to read them. The group is also standardising how conflicting preferences get resolved, which ad hoc files like llms.txt do not address.
