rfc9969v1.md		rfc9969.md
	---		---
	title: IAB AI-CONTROL Workshop Report		title: Report from the IAB Workshop on AI-CONTROL
	abbrev: IAB AI-CONTROL Workshop Report		abbrev: IAB AI-CONTROL Workshop Report
	category: info		category: info
	docname: draft-iab-ai-control-report-02		docname: draft-iab-ai-control-report-02
	number: 9969		number: 9969
	ipr: trust200902		ipr: trust200902
	submissiontype: IAB		submissiontype: IAB
	date: 2026-04		date: 2026-05
	obsoletes:		obsoletes:
	updates:		updates:
	consensus: true		consensus: true
	pi: [toc, symrefs, sortrefs]		pi: [toc, symrefs, sortrefs]
	v: 3		v: 3
	lang: en		lang: en
	keyword:		keyword:
	- policy		- policy
	- Artificial Intelligence		- Artificial Intelligence
	- Robots Exclusion Protocol		- Robots Exclusion Protocol
	skipping to change at line 93 ¶		skipping to change at line 93 ¶
	-		-
	ins: A. Lee		ins: A. Lee
	name: Ariel Lee		name: Ariel Lee
	-		-
	ins: C. Lund		ins: C. Lund
	name: Campbell Lund		name: Campbell Lund
	date: 2025		date: 2025
	--- abstract		--- abstract
	<!--[rfced] May we update the title to follow the format in other
	workshop reports?
	Original:
	IAB AI-CONTROL Workshop Report
	Perhaps:
	Report from the IAB Workshop on AI-CONTROL
	The AI-CONTROL Workshop was convened by the Internet Architecture Board (IAB) in Sept ember 2024. This report summarizes its significant points of discussion and identifie s topics that may warrant further consideration and work.		The AI-CONTROL Workshop was convened by the Internet Architecture Board (IAB) in Sept ember 2024. This report summarizes its significant points of discussion and identifie s topics that may warrant further consideration and work.
	Note that this document is a report on the proceedings of the workshop. The views an d positions documented in this report are those of the workshop participants and do n ot necessarily reflect IAB views and positions.		Note that this document is a report on the proceedings of the workshop. The views an d positions documented in this report are those of the workshop participants and do n ot necessarily reflect IAB views and positions.
	--- middle		--- middle
	# Introduction		# Introduction
	<!--[rfced] May we update the text as shown below (i.e., replace
	"large language models" with "Large Language Models (LLMs)", or would
	this update change the intended meaning?
	Original:
	The Internet is one of the major sources of data used to train
	large language models (Large Language Models (LLMs), or more
	generally, "Artificial Intelligence (AI)").
	Perhaps:
	The Internet is one of the major sources of data used to train
	Large Language Models (LLMs) (or, more generally, Artificial
	Intelligence (AI)).
	The Internet Architecture Board (IAB) holds occasional workshops designed to consider long-term issues and strategies for the Internet, and to suggest future directions f or the Internet architecture. This long-term planning function of the IAB is compleme ntary to the ongoing engineering efforts performed by working groups of the Internet Engineering Task Force (IETF).		The Internet Architecture Board (IAB) holds occasional workshops designed to consider long-term issues and strategies for the Internet, and to suggest future directions f or the Internet architecture. This long-term planning function of the IAB is compleme ntary to the ongoing engineering efforts performed by working groups of the Internet Engineering Task Force (IETF).
	The Internet is one of the major sources of data used to train large language models (Large Language Models (LLMs) or, more generally, Artificial Intelligence (AI)). Beca use this use was not envisioned by most publishers of information on the Internet, a means of expressing the owners' preferences regarding AI crawling has emerged, someti mes backed by law (e.g., in the European Union's AI Act {{AI-ACT}}).		The Internet is one of the major sources of data used to train Large Language Models (LLMs) (or, more generally, Artificial Intelligence (AI)). Because this use was not e nvisioned by most publishers of information on the Internet, a means of expressing th e owners' preferences regarding AI crawling has emerged, sometimes backed by law (e.g ., in the European Union's AI Act {{AI-ACT}}).
	The IAB convened the AI-CONTROL Workshop on 19-20 September 2024 to "explore practica l opt-out mechanisms for AI and build an understanding of use cases, requirements, an d other considerations in this space" {{CFP}}. In particular, the emerging practice o f using the Robots Exclusion Protocol {{?RFC9309}} -- also known as "robots.txt" -- h as not been coordinated between AI crawlers, resulting in considerable differences in how they treat it. Furthermore, robots.txt may or may not be a suitable way to contr ol AI crawlers. However, discussion was not limited to consideration of robots.txt, a nd approaches other than opt-out were considered.		The IAB convened the AI-CONTROL Workshop on 19-20 September 2024 to "explore practica l opt-out mechanisms for AI and build an understanding of use cases, requirements, an d other considerations in this space" {{CFP}}. In particular, the emerging practice o f using the Robots Exclusion Protocol {{?RFC9309}} -- also known as "robots.txt" -- h as not been coordinated between AI crawlers, resulting in considerable differences in how they treat it. Furthermore, robots.txt may or may not be a suitable way to contr ol AI crawlers. However, discussion was not limited to consideration of robots.txt, a nd approaches other than opt-out were considered.
	To ensure many viewpoints were represented, the program committee invited a broad sel ection of technical experts, AI vendors, content publishers, civil society advocates, and policymakers.		To ensure many viewpoints were represented, the program committee invited a broad sel ection of technical experts, AI vendors, content publishers, civil society advocates, and policymakers.
	## Chatham House Rule		## Chatham House Rule
	Participants agreed to conduct the workshop under the Chatham House Rule {{CHATHAM-HO USE}}, so this report does not attribute statements to individuals or organizations w ithout express permission. Most submissions to the workshop were public and thus attr ibutable; they are used here to provide substance and context.		Participants agreed to conduct the workshop under the Chatham House Rule {{CHATHAM-HO USE}}, so this report does not attribute statements to individuals or organizations w ithout express permission. Most submissions to the workshop were public and thus attr ibutable; they are used here to provide substance and context.
	{{attendees}} lists the workshop participants, unless they requested that this inform ation be withheld.		{{attendees}} lists the workshop participants, unless they requested that this inform ation be withheld.
	skipping to change at line 156 ¶		skipping to change at line 131 ¶
	# Workshop Scope and Discussion		# Workshop Scope and Discussion
	The workshop began by surveying the state of AI control.		The workshop began by surveying the state of AI control.
	Currently, Internet publishers express their preferences for how their content is tre ated for the purposes of AI training using a variety of mechanisms. These include dec larative mechanisms, such as terms of service, embedded metadata, and robots.txt {{RF C9309}}, as well as active mechanisms, such as use of paywalls and selective blocking of crawlers (e.g., by IP address or User-Agent).		Currently, Internet publishers express their preferences for how their content is tre ated for the purposes of AI training using a variety of mechanisms. These include dec larative mechanisms, such as terms of service, embedded metadata, and robots.txt {{RF C9309}}, as well as active mechanisms, such as use of paywalls and selective blocking of crawlers (e.g., by IP address or User-Agent).
	There was disagreement about the implications of AI opt-out overall. Research present ed at the workshop {{DECLINE}} indicates that the use of such controls is becoming mo re prevalent, reducing the availability of data to AI (for purposes including trainin g and inference-time usage). Some of the participants expressed concern about the imp lications of this -- although at least one AI vendor seemed less concerned by this, i ndicating that "there are plenty of tokens available" for training, even if many opt out. Others expressed a need to opt out of AI training because of how they perceive i ts effects on their control over content, seeing AI as usurping their relationships w ith customers and a potential threat to whole industries.		There was disagreement about the implications of AI opt-out overall. Research present ed at the workshop {{DECLINE}} indicates that the use of such controls is becoming mo re prevalent, reducing the availability of data to AI (for purposes including trainin g and inference-time usage). Some of the participants expressed concern about the imp lications of this -- although at least one AI vendor seemed less concerned by this, i ndicating that "there are plenty of tokens available" for training, even if many opt out. Others expressed a need to opt out of AI training because of how they perceive i ts effects on their control over content, seeing AI as usurping their relationships w ith customers and a potential threat to whole industries.
	However, there was quick agreement that both viewpoints were harmed by the current st ate of AI opt-out -- a situation where "no one is better off" (in the words of one pa rticipant).		However, there was quick agreement that both viewpoints were harmed by the current st ate of AI opt-out -- a situation where "no one is better off" (in the words of one pa rticipant).
	<!--[rfced] In the last sentence below, please clarify what "both"		Much of that dysfunction was attributed to the lack of coordination and standards for
	refers to - is it new vendors and policy updates?		AI opt-out. Currently, content publishers need to consult with each AI vendor to und
			erstand how to opt out of training their products, as there is significant variance i
	Current:		n each vendor's behavior. Furthermore, publishers need to continually monitor both ne
	Much of that dysfunction was attributed to the lack of coordination		w vendors and policy updates from the vendors they are aware of.
	and standards for AI opt-out. Currently, content publishers need to
	consult with each AI vendor to understand how to opt out of training
	their products, as there is significant variance in each vendor's
	behavior. Furthermore, publishers need to continually monitor both for
	new vendors and changes to the policies of the vendors they are
	aware of.
	Perhaps:
	... Furthermore, publishers need to continually monitor both new
	vendors and policy updates from the vendors they are aware
	of.
	Much of that dysfunction was attributed to the lack of coordination and standards for
	AI opt-out. Currently, content publishers need to consult with each AI vendor to und
	erstand how to opt out of training their products, as there is significant variance i
	n each vendor's behavior. Furthermore, publishers need to continually monitor for bot
	h new vendors and changes to the policies of the vendors they are aware of.
	Underlying those immediate issues, however, are significant constraints that could be attributed to uncertainties in the legal context, the nature of AI, and the implicat ions of needing to opt out of crawling for it.		Underlying those immediate issues, however, are significant constraints that could be attributed to uncertainties in the legal context, the nature of AI, and the implicat ions of needing to opt out of crawling for it.
	## Crawl Time vs. Inference Time		## Crawl Time vs. Inference Time
	Perhaps most significant is the "crawl time vs. inference time" problem. Statements o f preference are apparent at crawl time, bound to content either by location (e.g., r obots.txt) or embedded inside the content itself as metadata. However, the target of those directives is often disassociated from the crawler, either because the crawl da ta is not only used for training AI models or because the preferences could be applic able at inference time.		Perhaps most significant is the "crawl time vs. inference time" problem. Statements o f preference are apparent at crawl time, bound to content either by location (e.g., r obots.txt) or embedded inside the content itself as metadata. However, the target of those directives is often disassociated from the crawler, either because the crawl da ta is not only used for training AI models or because the preferences could be applic able at inference time.
	### Multiple Uses for Crawl Data		### Multiple Uses for Crawl Data
	A crawl's data might have multiple uses because the vendor also has another product t hat uses it (e.g., a search engine) or because the crawl is performed by a party othe r than the AI vendor. Both are very common patterns: Operators of many Internet searc h engines also train AI models, and many AI models use third-party crawl data. In eit her case, conflating different uses can change the incentives for publishers to coope rate with the crawler.		A crawl's data might have multiple uses because the vendor also has another product t hat uses it (e.g., a search engine) or because the crawl is performed by a party othe r than the AI vendor. Both are very common patterns: Operators of many Internet searc h engines also train AI models, and many AI models use third-party crawl data. In eit her case, conflating different uses can change the incentives for publishers to coope rate with the crawler.
	skipping to change at line 204 ¶		skipping to change at line 161 ¶
	When data is used to train an LLM, the resulting model does not have the ability to o nly selectively use a portion of it when performing a task because inference uses the whole model, and it is not possible to identify specific input data for its use in d oing so.		When data is used to train an LLM, the resulting model does not have the ability to o nly selectively use a portion of it when performing a task because inference uses the whole model, and it is not possible to identify specific input data for its use in d oing so.
	This means that while publishers' preferences may be available when content is crawle d, they generally are not when inference takes place. Those preferences that are stat ed in reference to use by AI -- for example, "no military uses" or "non-commercial on ly" -- cannot be applied by a general-purpose "foundation" model.		This means that while publishers' preferences may be available when content is crawle d, they generally are not when inference takes place. Those preferences that are stat ed in reference to use by AI -- for example, "no military uses" or "non-commercial on ly" -- cannot be applied by a general-purpose "foundation" model.
	This leaves a few unappealing choices to AI vendors that wish to comply with those pr eferences. They can simply omit such data from foundation models, thereby reducing th eir viability. Or they can create a separate model for each permutation of preference s -- with a likely proliferation of models as the set of permutations expands.		This leaves a few unappealing choices to AI vendors that wish to comply with those pr eferences. They can simply omit such data from foundation models, thereby reducing th eir viability. Or they can create a separate model for each permutation of preference s -- with a likely proliferation of models as the set of permutations expands.
	Compounding this issue was the observation that preferences change over time, whereas LLMs are created over long time frames and cannot easily be updated to reflect those changes. Of particular concern to some was how this makes an opt-out regime "stickie r" because content that has no associated preference (such as that which predates the authors' knowledge of LLMs) is allowed to be used for these unforeseen purposes.		Compounding this issue was the observation that preferences change over time, whereas LLMs are created over long time frames and cannot easily be updated to reflect those changes. Of particular concern to some was how this makes an opt-out regime "stickie r" because content that has no associated preference (such as that which predates the authors' knowledge of LLMs) is allowed to be used for these unforeseen purposes.
	## Trust		## Trust
	<!--[rfced] May we update "was felt by participants to contribute to"		Participants felt that the disconnection between the statement of preferences and its
	as shown below for easier readability?		application contribute to a lack of trust in the ecosystem, along with the typical l
			ack of attribution for data sources in LLMs, a lack of an incentive for publishers to
	Original:		contribute data, and finally (and most noted) a lack of any means of monitoring comp
	This disconnection between the statement of preferences and its		liance with preferences.
	application was felt by participants to contribute to a lack of
	trust in the ecosystem, along with the typical lack of attribution
	for data sources in LLMs, lack of an incentive for publishers to
	contribute data, and finally (and most noted) a lack of any means
	of monitoring compliance with preferences.
	Perhaps:
	Participants felt that the disconnection between the statement of
	preferences and its application contributes to a lack of trust in
	the ecosystem, along with the typical lack of attribution for data
	sources in LLMs, a lack of an incentive for publishers to
	contribute data, and finally (and most noted) a lack of any means
	of monitoring compliance with preferences.
	This disconnection between the statement of preferences and its application was felt
	by participants to contribute to a lack of trust in the ecosystem, along with the typ
	ical lack of attribution for data sources in LLMs, lack of an incentive for publisher
	s to contribute data, and finally (and most noted) lack of any means of monitoring co
	mpliance with preferences.
	This lack of trust led some participants to question whether communicating preference s is sufficient in all cases without an accompanying way to enforce them, or even to audit adherence to them. Some participants also indicated that a lack of trust was th e primary cause of the increasingly prevalent blocking of AI crawler IP addresses, am ong other measures.		This lack of trust led some participants to question whether communicating preference s is sufficient in all cases without an accompanying way to enforce them, or even to audit adherence to them. Some participants also indicated that a lack of trust was th e primary cause of the increasingly prevalent blocking of AI crawler IP addresses, am ong other measures.
	## Attachment		## Attachment
	One of the primary focuses of the workshop was on _attachment_, i.e., how preferences are associated with content on the Internet. A range of mechanisms was discussed.		One of the primary focuses of the workshop was on _attachment_, i.e., how preferences are associated with content on the Internet. A range of mechanisms was discussed.
	### robots.txt (and Similar)		### robots.txt (and Similar)
	The Robots Exclusion Protocol {{RFC9309}} is widely recognized by AI vendors as an at tachment mechanism for preferences. Several deficiencies were discussed.		The Robots Exclusion Protocol {{RFC9309}} is widely recognized by AI vendors as an at tachment mechanism for preferences. Several deficiencies were discussed.
	skipping to change at line 250 ¶		skipping to change at line 187 ¶
	If content is copied or moved to a different site, the preferences at the new site ne ed to be explicitly transferred because robots.txt is a separate resource.		If content is copied or moved to a different site, the preferences at the new site ne ed to be explicitly transferred because robots.txt is a separate resource.
	These deficiencies led many participants to feel that robots.txt cannot be the only s olution to opt-out: Rather, it should be part of a larger system that addresses its s hortcomings.		These deficiencies led many participants to feel that robots.txt cannot be the only s olution to opt-out: Rather, it should be part of a larger system that addresses its s hortcomings.
	Participants noted that other similar attachment mechanisms have been proposed. Howev er, none appear to have gained as much attention or implementation (both by AI vendor s and content owners) as robots.txt.		Participants noted that other similar attachment mechanisms have been proposed. Howev er, none appear to have gained as much attention or implementation (both by AI vendor s and content owners) as robots.txt.
	### Embedding		### Embedding
	Another mechanism for associating preferences with content is to embed them into the content itself. Many formats used on the Internet allow this; for example, HTML has t he `<meta>` tag, images have Extensible Metadata Platform (XMP) and similar metadata sections, and XML and JSON have rich potential for extensions to carry such data.		Another mechanism for associating preferences with content is to embed them into the content itself. Many formats used on the Internet allow this; for example, HTML has t he `<meta>` tag, images have Extensible Metadata Platform (XMP) and similar metadata sections, and XML and JSON have rich potential for extensions to carry such data.
	<!--[rfced] Is "when it is moved" referring to "preferences"? If yes,		Embedded preferences were seen to have the advantage of granularity, and of "travelin
	may we update the text as follows?		g with" content as it is produced, when the content that embeds the preferences is mo
			ved from site to site or when it is stored offline.
	Original:
	Embedded preferences were seen to have the advantage of granularity,
	and of "travelling with" content as it is produced, when it is moved
	from site to site, or when it is stored offline.
	Perhaps:
	Embedded preferences were seen to have the advantage of granularity,
	and of "traveling with" content as it is produced, when they are moved
	from site to site or when they are stored offline.
	Embedded preferences were seen to have the advantage of granularity, and of "travelin
	g with" content as it is produced, when it is moved from site to site or when it is s
	tored offline.
	However, several participants pointed out that embedded preferences are easily stripp ed from most formats. This is a common practice for reducing the size of a file (ther eby improving performance when downloading it) and for assuring privacy (since metada ta often leaks information unintentionally).		However, several participants pointed out that embedded preferences are easily stripp ed from most formats. This is a common practice for reducing the size of a file (ther eby improving performance when downloading it) and for assuring privacy (since metada ta often leaks information unintentionally).
	Furthermore, some types of content are not suitable for embedding. For example, it is not possible to embed preferences into purely textual content, and web pages with co ntent from several producers (such as a social media or comment feeds) cannot easily reflect preferences for each one.		Furthermore, some types of content are not suitable for embedding. For example, it is not possible to embed preferences into purely textual content, and web pages with co ntent from several producers (such as a social media or comment feeds) cannot easily reflect preferences for each one.
	Participants noted that the means of embedding preferences in many formats would need to be determined by or coordinated with organizations outside the IETF. For example, HTML and many image formats are maintained by external bodies.		Participants noted that the means of embedding preferences in many formats would need to be determined by or coordinated with organizations outside the IETF. For example, HTML and many image formats are maintained by external bodies.
	### Registries		### Registries
	In some existing copyright management regimes, it is already common to have a registr y of works that is consulted upon use. For example, this approach is often used for p hotographs, music, and video.		In some existing copyright management regimes, it is already common to have a registr y of works that is consulted upon use. For example, this approach is often used for p hotographs, music, and video.
	skipping to change at line 439 ¶		skipping to change at line 362 ¶
	# Acknowledgements		# Acknowledgements
	{:numbered="false"}		{:numbered="false"}
	The program committee and the IAB would like to thank Wilkinson Barker Knauer for the ir generosity in hosting the workshop.		The program committee and the IAB would like to thank Wilkinson Barker Knauer for the ir generosity in hosting the workshop.
	We also thank our scribes for capturing notes that assisted in the production of this report:		We also thank our scribes for capturing notes that assisted in the production of this report:
	* {{{Zander Arnao}}}		* {{{Zander Arnao}}}
	* {{{Andrea Dean}}}		* {{{Andrea Dean}}}
	* {{{Patrick Yurky}}}		* {{{Patrick Yurky}}}
	<!-- [rfced] FYI - We have added expansions for the following abbreviations
	per Section 3.6 of RFC 7322 ("RFC Style Guide"). Please review each
	expansion in the document carefully to ensure correctness.
	Standards Development Organization (SDO)
	Extensible Metadata Platform (XMP)
	<!-- [rfced] Please review the "Inclusive Language" portion of the online
	Style Guide <https://www.rfc-editor.org/styleguide/part2/#inclusive_language>
	and let us know if any changes are needed. Updates of this nature typically
	result in more precise language, which is helpful for readers.
	Note that our script did not flag any words in particular, but this should
	still be reviewed as a best practice.
End of changes. 9 change blocks.
	88 lines changed or deleted		16 lines changed or added
This html diff was produced by rfcdiff 1.48.