In Situ Operations, Administration, and Maintenance (IOAM) Direct ExportingFuturewei2330 Central ExpresswaySanta Clara95050United States of Americahaoyu.song@futurewei.comNvidiaSuite 100350 Oakmead ParkwaySunnyvaleCA94085United States of Americagbarak@nvidia.comCisco Systems, Inc.Hansaallee 249Duesseldorf40549Germanyfbrockne@cisco.comThoughtspot3rd Floor, Indiqube Orion, Garden Layout, HSR Layout24th Main RdBangaloreKarnataka560 102Indiashwetha.bhandari@thoughtspot.comHuawei8-2 MatamHaifa3190501Israeltal.mizrahi.phd@gmail.com
tsv
ippmIOAMTelemetryIn situ Operations, Administration, and Maintenance (IOAM) is used
for recording and collecting operational and telemetry information.
Specifically, IOAM allows telemetry data to be pushed into data packets
while they traverse the network. This document introduces a new IOAM
option type (denoted IOAM-Option-Type) called the "IOAM Direct Export (DEX) Option-Type". This Option-Type is used as a trigger for IOAM data to be directly
exported or locally aggregated without being pushed into in-flight data
packets. The exporting method and format are outside the scope of this
document.Status of This Memo
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by
the Internet Engineering Steering Group (IESG). Further
information on Internet Standards is available in Section 2 of
RFC 7841.
Information about the current status of this document, any
errata, and how to provide feedback on it may be obtained at
.
Copyright Notice
Copyright (c) 2022 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
() in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Revised BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Revised BSD License.
Table of Contents
. Introduction
. Conventions
. Requirements Language
. Terminology
. The Direct Exporting (DEX) IOAM-Option-Type
. Overview
. DEX Packet Selection
. Responding to the DEX Trigger
. The DEX Option-Type Format
. IANA Considerations
. IOAM Type
. IOAM DEX Flags
. IOAM DEX Extension-Flags
. Performance Considerations
. Security Considerations
. References
. Normative References
. Informative References
. Notes about the History of This Document
Acknowledgments
Contributors
Authors' Addresses
IntroductionIOAM is used for monitoring traffic in the
network and for incorporating IOAM data fields (denoted
IOAM-Data-Fields) into in-flight data packets.IOAM makes use of four possible IOAM-Option-Types, defined in : Pre-allocated Trace, Incremental Trace, Proof of Transit (POT), and Edge-to-Edge.This document defines a new IOAM-Option-Type called the "IOAM Direct Export (DEX)
Option-Type". This Option-Type is used as a trigger for IOAM nodes
to locally aggregate and process IOAM data and/or to export it to a
receiving entity (or entities). Throughout the document, this
functionality is referred to as "collection" and/or "exporting". In this context, a
"receiving entity" is an entity
that resides within the IOAM domain such as a collector, analyzer,
controller, decapsulating node, or software module in one of the IOAM nodes.Note that even though the IOAM-Option-Type is called "Direct Export",
it depends on the deployment whether the receipt of a packet with a DEX
Option-Type leads to the creation of another packet. Some deployments
might simply use the packet with the DEX Option-Type to trigger local
processing of Operations, Administration, and Maintenance (OAM) data. The functionality of this local processing is
not within the scope of this document.ConventionsRequirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as
described in BCP 14
when, and only when, they appear in all capitals, as shown here.
TerminologyAbbreviations used in this document:
IOAM:
In situ Operations, Administration, and
Maintenance
OAM:
Operations, Administration, and Maintenance
DEX:
Direct Exporting
The Direct Exporting (DEX) IOAM-Option-TypeOverviewThe DEX Option-Type is used as a trigger for collecting IOAM data
locally or exporting it to a receiving entity (or entities).
Specifically, the DEX Option-Type can be used as a trigger for
collecting IOAM data by an IOAM node and locally aggregating it; thus,
this aggregated data can be periodically pushed to a receiving entity
or pulled by a receiving entity on-demand.This Option-Type is incorporated into data packets by an IOAM
encapsulating node and removed by an IOAM decapsulating node, as
illustrated in . The Option-Type can be read,
but not modified, by transit nodes. Note that the terms "IOAM encapsulating node",
"IOAM decapsulating node", and "IOAM transit node" are as defined in .The DEX Option-Type is used as a trigger to collect and/or export
IOAM data. The trigger applies to transit nodes, the decapsulating
node, and the encapsulating node:
An IOAM encapsulating node configured to incorporate the DEX
Option-Type encapsulates the packets (or possibly a subset of the
packets) it forwards with the DEX Option-Type and MAY export and/or collect
the requested IOAM data immediately. Only IOAM encapsulating nodes
are allowed to add the DEX Option-Type to a packet. An IOAM
encapsulating node can generate probe packets that incorporate the
DEX Option-Type. These probe packets can be generated periodically
or on-demand (for example, triggered by the management plane). The
specification of such probe packets is outside the scope of this
document.
A transit node that processes a packet with the DEX Option-Type
MAY export and/or collect the requested IOAM data.
An IOAM decapsulating node that processes a packet with the DEX
Option-Type MAY export and/or collect the requested IOAM data and
MUST decapsulate the IOAM header.
As in , the DEX Option-Type can be
incorporated into all or a subset of the traffic that is forwarded by
the encapsulating node, as further discussed in . Moreover, IOAM nodes respond to the DEX
trigger by exporting and/or collecting IOAM data either for all
traversing packets that carry the DEX Option-Type or selectively only
for a subset of these packets, as further discussed in .DEX Packet SelectionIf an IOAM encapsulating node incorporates the DEX Option-Type
into all the traffic it forwards, it may lead to an excessive amount
of exported data, which may overload the network and the receiving
entity. Therefore, an IOAM encapsulating node that supports the DEX
Option-Type MUST support the ability to incorporate the DEX
Option-Type selectively into a subset of the packets that are
forwarded by the IOAM encapsulating node.Various methods of packet selection and sampling have been
previously defined, such as and . Similar techniques can be applied by an IOAM
encapsulating node to apply DEX to a subset of the forwarded
traffic.The subset of traffic that is forwarded or transmitted with a DEX
Option-Type SHOULD NOT exceed 1/N of the interface capacity on any
of the IOAM encapsulating node's interfaces. It is noted that this
requirement applies to the total traffic that incorporates a DEX
Option-Type, including traffic that is forwarded by the IOAM
encapsulating node and probe packets that are generated by the IOAM
encapsulating node. In this context, N is a parameter that can be
configurable by network operators. If there is an upper bound, M, on
the number of IOAM transit nodes in any path in the network, then it
is RECOMMENDED to use an N such that N >> M (i.e., N is much greater
than M). The rationale is
that a packet that includes a DEX Option-Type may trigger an
exported packet from each IOAM transit node along the path for a
total of M exported packets. Thus, if N >> M, then the number
of exported packets is significantly lower than the number of data
packets forwarded by the IOAM encapsulating node. If there is no
prior knowledge about the network topology or size, it is
RECOMMENDED to use N>100.Responding to the DEX TriggerThe DEX Option-Type specifies which IOAM-Data-Fields should be
exported and/or collected, as specified in . As mentioned above, the data can be locally collected, aggregated, and/or
exported to a receiving entity proactively or on-demand. If IOAM data is
exported, the format and encapsulation of the packet that contains
the exported data is not within the scope of the current document.
For example, the export format can be based on .An IOAM node that performs DEX-triggered exporting MUST support
the ability to limit the rate of the exported packets. The rate of
exported packets SHOULD be limited so that the number of exported
packets is significantly lower than the number of packets that are
forwarded by the device. The exported data rate SHOULD NOT exceed
1/N of the interface capacity on any of the IOAM node's interfaces.
It is RECOMMENDED to use N>100. Depending on the IOAM node's
architecture considerations, the export rate may be limited to a
lower number in order to avoid loading the IOAM node. An IOAM node
MAY maintain a counter or a set of counters that count the events in
which the IOAM node receives a packet with the DEX Option-Type and
does not collect and/or export data due to the rate limits.IOAM nodes SHOULD NOT be configured to export packets
over a path or a tunnel
that is subject to IOAM direct exporting. Furthermore, IOAM
encapsulating nodes that can identify a packet as an IOAM exported
packet MUST NOT push a DEX Option-Type into such a packet. This
requirement is intended to prevent nested exporting and/or exporting
loops.A transit or decapsulating IOAM node that receives an unknown
IOAM-Option-Type ignores it (as defined in ); specifically, nodes that do not support the
DEX Option-Type ignore it. As per , note that
a decapsulating node removes the IOAM encapsulation and all its
IOAM-Option-Types. Specifically, this applies to the case where one of these
options is a (possibly unknown) DEX Option-Type. The ability to skip
over a (possibly unknown) DEX Option-Type in the parsing or in the
decapsulation procedure is dependent on the specific encapsulation,
which is outside the scope of this document. For example, when IOAM
is encapsulated in IPv6 , the DEX Option-Type is
incorporated either in a Hop-by-Hop options header or in a
Destination options header; thus, it can be skipped using the length
field in the options header.The DEX Option-Type FormatThe format of the DEX Option-Type is depicted in . The length of the DEX Option-Type is at least
8 octets. The DEX Option-Type MAY include one or more optional fields.
The existence of the optional fields is indicated by the corresponding
flags in the Extension-Flags field. Two optional fields are defined in
this document: the Flow ID and Sequence Number fields. Every
optional field MUST be exactly 4 octets long. Thus, the
Extension-Flags field explicitly indicates the length of the DEX
Option-Type. Defining a new optional field requires an allocation of a
corresponding flag in the Extension-Flags field, as specified in .
Namespace-ID:
A 16-bit identifier of the IOAM
namespace, as defined in .
Flags:
An 8-bit field, comprised of 8 1-bit
subfields. Flags are allocated by IANA, as defined in .
Extension-Flags:
An 8-bit field, comprised of 8
1-bit subfields. Extension-Flags are allocated by IANA, as
defined in . Every bit in the
Extension-Flag field that is set to 1 indicates the existence of a
corresponding optional 4-octet field. An IOAM node that receives a
DEX Option-Type with an unknown flag set to 1 MUST ignore the
corresponding optional field.
IOAM-Trace-Type:
A 24-bit identifier that specifies
which IOAM-Data-Fields should be exported. The format of this
field is as defined in . Specifically, the
bit that corresponds to the Checksum Complement IOAM-Data-Field
SHOULD be assigned to be zero by the IOAM encapsulating node and
ignored by transit and decapsulating nodes. The reason for this is
that the Checksum Complement is intended for in-flight packet
modifications and is not relevant for direct exporting.
Reserved:
This field MUST be ignored by the
receiver.
Optional fields:
The optional fields, if present,
reside after the Reserved field. The order of the optional fields
is according to the order of the respective bits, starting from
the most significant bit, that are enabled in the
Extension-Flags field. Each optional field is 4 octets long.
Flow ID:
An optional 32-bit field representing the
flow identifier. If the actual Flow ID is shorter than 32 bits, it
is zero padded in its most significant bits. The field is set at
the encapsulating node. The Flow ID can be used to correlate the
exported data of the same flow from multiple nodes and from
multiple packets. Flow ID values are expected to be allocated in a
way that avoids collisions. For example, random assignment of Flow
ID values can be subject to collisions, while
centralized allocation can avoid this problem. The specification
of the Flow ID allocation method is not within the scope of this
document.
Sequence Number:
An optional 32-bit sequence number
starting from 0 and incremented by 1 for each
packet from the same flow at the encapsulating node that includes
the DEX option. The Sequence
Number, when combined with the Flow ID, provides a convenient
approach to correlate the exported data from the same user
packet.
IANA ConsiderationsIOAM TypeThe "IOAM Option-Type" registry is defined in . IANA has allocated the following code
point from the "IOAM Option-Type" registry as follows:
Code Point:
4
Name
IOAM Direct Export (DEX) Option-Type
Description:
Direct exporting
Reference:
This document
IOAM DEX FlagsIANA has created the "IOAM DEX Flags" registry. This
registry includes 8 flag bits. Allocation is based on the "IETF
Review" procedure defined in .New registration requests MUST use the following template:
Bit:
Desired bit to be allocated in the 8-bit Flags
field of the DEX Option-Type.
Description:
Brief description of the newly
registered bit.
Reference:
Reference to the document that defines
the new bit.
IOAM DEX Extension-FlagsIANA has created the "IOAM DEX Extension-Flags" registry.
This registry includes 8 flag bits. Bit 0 (the most significant bit)
and bit 1 in the registry are allocated by this document and
described in . Allocation of the other bits
should be performed based on the "IETF Review" procedure defined
in .
Bit 0:
"Flow ID [RFC9326]"
Bit 1:
"Sequence Number [RFC9326]"
New registration requests MUST use the following template:
Bit:
Desired bit to be allocated in the 8-bit
Extension-Flags field of the DEX Option-Type.
Description:
Brief description of the newly
registered bit.
Reference:
Reference to the document that defines
the new bit.
Performance ConsiderationsThe DEX Option-Type triggers IOAM data to be collected and/or
exported packets to be exported to a receiving entity (or entities). In
some cases, this may impact the receiving entity's performance or the
performance along the paths leading to it.Therefore, the performance impact of these exported packets is
limited by taking two measures: at the encapsulating nodes by selective
DEX encapsulation () and at the transit
nodes by limiting exporting rate (). These
two measures ensure that direct exporting is used at a rate that does
not significantly affect the network bandwidth and does not overload
the receiving entity. Moreover, it is possible to load balance the
exported data among multiple receiving entities, although the exporting
method is not within the scope of this document.It should be noted that, in some networks, DEX data may be exported
over an out-of-band network in which a large volume of exported traffic
does not compromise user traffic. In this case, an operator may choose to
disable the exporting rate limiting.Security ConsiderationsThe security considerations of IOAM in general are discussed in . Specifically, an attacker may try to use the
functionality that is defined in this document to attack the
network.An attacker may attempt to overload network devices by injecting
synthetic packets that include the DEX Option-Type. Similarly, an
on-path attacker may maliciously incorporate the DEX Option-Type into
transit packets or maliciously remove it from packets in which it is
incorporated.Forcing DEX, either in synthetic packets or in transit packets, may
overload the IOAM nodes and/or the receiving entity (or entities). Since
this mechanism affects multiple devices along the network path, it
potentially amplifies the effect on the network bandwidth, the
storage of the devices that collect the data, and the receiving
entity's load.The amplification effect of DEX may be worse in wide area networks in
which there are multiple IOAM-Domains. For example, if DEX is used in
IOAM-Domain 1 for exporting IOAM data to a receiving entity, then the
exported packets of IOAM-Domain 1 can be forwarded through IOAM-Domain 2, in
which they are subject to DEX. In turn, the exported packets of IOAM-Domain 2 may be forwarded through another IOAM domain (or through IOAM-Domain 1);
theoretically, this recursive amplification may continue infinitely.In order to mitigate the attacks described above, the following
requirements () have been defined:
Selective DEX () is applied by IOAM
encapsulating nodes in order to limit the potential impact of DEX
attacks to a small fraction of the traffic.
Rate limiting of exported traffic () is
applied by IOAM nodes in order to prevent overloading attacks and
to significantly limit the scale of amplification attacks.
IOAM encapsulating nodes are required to avoid pushing the DEX
Option-Type into IOAM exported packets (),
thus preventing some of the amplification and export loop
scenarios.
Although the exporting method is not within the scope of this
document, any exporting method MUST secure the exported data from the
IOAM node to the receiving entity in order to protect the
confidentiality and guarantee the integrity of the exported data.
Specifically, an IOAM node that
performs DEX exporting MUST send the exported data to a pre-configured
trusted receiving entity that is in the same IOAM-Domain as the exporting
IOAM node.
Furthermore, an IOAM node MUST gain explicit
consent to export data to a receiving entity before starting to send
exported data.An attacker may keep track of the information sent in DEX headers as
a means of reconnaissance. This form of recon can be mitigated to some
extent by careful allocation of the Flow ID and Sequence Number space
in a way that does not compromise privacy aspects, such as customer
identities.The integrity of the DEX Option-Type can be protected through a
mechanism of the encapsulating protocol. While introduces an integrity
protection mechanism that protects the integrity of IOAM-Data-Fields,
the DEX Option-Type does not include IOAM-Data-Fields; therefore,
these integrity protection mechanisms are not applicable to the DEX
Option-Type. As discussed in the threat analysis of , injection or modification
of IOAM-Option-Type headers are threats that are not addressed in
IOAM.IOAM is assumed to be deployed in a restricted administrative domain,
thus limiting the scope of the threats above and their effect. This is a
fundamental assumption with respect to the security aspects of IOAM, as
further discussed in .ReferencesNormative ReferencesKey words for use in RFCs to Indicate Requirement LevelsIn many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.Ambiguity of Uppercase vs Lowercase in RFC 2119 Key WordsRFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)In situ Operations, Administration, and Maintenance (IOAM) collects operational and telemetry information in the packet while the packet traverses a path between two points in the network. This document discusses the data fields and associated data types for IOAM. IOAM-Data-Fields can be encapsulated into a variety of protocols, such as Network Service Header (NSH), Segment Routing, Generic Network Virtualization Encapsulation (Geneve), or IPv6. IOAM can be used to complement OAM mechanisms based on, e.g., ICMP or other types of probe packets.Informative ReferencesIntegrity of In-situ OAM Data FieldsCisco Systems, Inc.ThoughtspotHuaweiUniversite de Liege In-situ Operations, Administration, and Maintenance (IOAM) records
operational and telemetry information in the packet while the packet
traverses a path in the network. IETF protocols require features to
ensure their security. This document describes the integrity
protection of IOAM-Data-Fields.
Work in ProgressIn-situ OAM IPv6 OptionsThoughtspotCisco Systems, Inc. In-situ Operations, Administration, and Maintenance (IOAM) records
operational and telemetry information in the packet while the packet
traverses a path between two points in the network. This document
outlines how IOAM data fields are encapsulated in IPv6.
Work in ProgressIn-situ OAM raw data export with IPFIXBarefoot Networks, an Intel companyCisco Systems, Inc.ThoughtspotCisco Systems, Inc. In-situ Operations, Administration, and Maintenance (IOAM) records
operational and telemetry information in the packet while the packet
traverses a path between two points in the network. This document
discusses how In-situ Operations, Administration, and Maintenance
(IOAM) information can be exported in raw, i.e. uninterpreted, format
from network devices to systems, such as monitoring or analytics
systems using IPFIX.
Work in ProgressMarking-based Direct Export for On-path TelemetryFuturewei TechnologiesEricssonCisco Systems, Inc.Cisco Systems, Inc.HuaweiHuaweiSwisscomVerizon Inc.SK TelecomLG U+ The document describes a postcard-based telemetry method using
packet-marking, referred to as PBT-M. PBT-M does not carry the
telemetry data in user packets but sends the telemetry data through a
dedicated packet. PBT-M does not require an extra instruction header
but claims a bit as a trigger in existing header fields. PBT-M
raises some unique issues that need to be considered. This document
describes the high level scheme and covers the common requirements
and issues when applying PBT-M in different networks. PBT-M is
complementary to the other on-path telemetry schemes such as IOAM.
Work in ProgressSampling and Filtering Techniques for IP Packet SelectionThis document describes Sampling and Filtering techniques for IP packet selection. It provides a categorization of schemes and defines what parameters are needed to describe the most common selection schemes. Furthermore, it shows how techniques can be combined to build more elaborate packet Selectors. The document provides the basis for the definition of information models for configuring selection techniques in Metering Processes and for reporting the technique in use to a Collector. [STANDARDS-TRACK]Guidelines for the Use of the "OAM" Acronym in the IETFAt first glance, the acronym "OAM" seems to be well-known and well-understood. Looking at the acronym a bit more closely reveals a set of recurring problems that are revisited time and again.This document provides a definition of the acronym "OAM" (Operations, Administration, and Maintenance) for use in all future IETF documents that refer to OAM. There are other definitions and acronyms that will be discussed while exploring the definition of the constituent parts of the "OAM" term. This memo documents an Internet Best Current Practice.Flow Selection TechniquesThe Intermediate Flow Selection Process is the process of selecting a subset of Flows from all observed Flows. The Intermediate Flow Selection Process may be located at an IP Flow Information Export (IPFIX) Exporter or Collector, or within an IPFIX Mediator. It reduces the effort of post-processing Flow data and transferring Flow Records. This document describes motivations for using the Intermediate Flow Selection process and presents Intermediate Flow Selection techniques. It provides an information model for configuring Intermediate Flow Selection Process techniques and discusses what information about an Intermediate Flow Selection Process should be exported.Guidelines for Writing an IANA Considerations Section in RFCsMany protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA).To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry.This is the third edition of this document; it obsoletes RFC 5226.In Situ Operations, Administration, and Maintanence (IOAM) Loopback and Active FlagsNotes about the History of This DocumentThis document evolved from combining some of the concepts of PBT-I
from with
immediate exporting from early versions of
.In order to help correlate and order the exported packets, it is
possible to include the Hop_Lim/Node_ID IOAM-Data-Field in exported
packets. If the IOAM-Trace-Type has the
Hop_Lim/Node_ID bit set, then exported packets include the
Hop_Lim/Node_ID IOAM-Data-Field, which contains the TTL/Hop Limit value
from a lower layer protocol.
An alternative approach was considered during the design of this
document, according to which a 1-octet Hop Count field would be included
in the DEX header (presumably by claiming some space from the Flags
field). The Hop Limit would start from 0 at the encapsulating node and
be incremented by each IOAM transit node that supports the DEX
Option-Type. In this approach, the Hop Count field value would also be
included in the exported packet.AcknowledgmentsThe authors thank , , ,
s, , , , , and other members of the IPPM
working group for many helpful comments.ContributorsThe Editors would like to recognize the contributions of the
following individuals to this document.Huawei156 Beiqing Rd.Beijing100095Chinazhoutianran@huawei.comHuawei156 Beiqing Rd.Beijing100095Chinalizhenbin@huawei.comCisco Systems, Inc.170 West Tasman Dr.San JoseCA95134United States of Americasramesh@cisco.comAuthors' AddressesFuturewei2330 Central ExpresswaySanta Clara95050United States of Americahaoyu.song@futurewei.comNvidiaSuite 100350 Oakmead ParkwaySunnyvaleCA94085United States of Americagbarak@nvidia.comCisco Systems, Inc.Hansaallee 249Duesseldorf40549Germanyfbrockne@cisco.comThoughtspot3rd Floor, Indiqube Orion, Garden Layout, HSR Layout24th Main RdBangaloreKarnataka560 102Indiashwetha.bhandari@thoughtspot.comHuawei8-2 MatamHaifa3190501Israeltal.mizrahi.phd@gmail.com