Deterministic Networking ArchitectureHuawei3101 Rio WaySpring ValleyCalifornia91977United States of America+1 925 980 6430nfinn@nfinnconsulting.comCisco SystemsVillage d'Entreprises Green Side400, Avenue de RoumanilleBatiment T3Biot - Sophia Antipolis06410France+33 4 97 23 26 34pthubert@cisco.comEricssonMagyar tudosok korutja 11BudapestHungary1117balazs.a.varga@ericsson.comEricssonMagyar tudosok korutja 11BudapestHungary1117janos.farkas@ericsson.com
Internet
DetNetTSN, Bounded Lantency, Reliable Networking, Available Networking
This document provides the overall architecture for
Deterministic Networking (DetNet), which provides a capability
to carry specified unicast or multicast data flows for
real-time applications with extremely low data loss rates and
bounded latency within a network domain. Techniques used
include 1) reserving data-plane resources for individual (or
aggregated) DetNet flows in some or all of the intermediate
nodes along the path of the flow, 2) providing explicit routes
for DetNet flows that do not immediately change with the
network topology, and 3) distributing data from DetNet flow
packets over time and/or space to ensure delivery of each
packet's data in spite of the loss of a path. DetNet operates
at the IP layer and delivers service over lower-layer
technologies such as MPLS and Time-Sensitive
Networking (TSN) as defined by IEEE 802.1.
Status of This Memo
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by
the Internet Engineering Steering Group (IESG). Further
information on Internet Standards is available in Section 2 of
RFC 7841.
Information about the current status of this document, any
errata, and how to provide feedback on it may be obtained at
.
Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
() in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents
. Introduction
. Terminology
. Terms Used in This Document
. Dictionary of Terms Used by TSN and DetNet
. Providing the DetNet Quality of Service
. Primary Goals Defining the DetNet QoS
. Mechanisms to Achieve DetNet QoS
. Resource Allocation
. Service Protection
. Explicit Routes
. Secondary Goals for DetNet
. Coexistence with Normal Traffic
. Fault Mitigation
. DetNet Architecture
. DetNet Stack Model
. Representative Protocol Stack Model
. DetNet Data-Plane Overview
. Network Reference Model
. DetNet Systems
. End System
. DetNet Edge, Relay, and Transit Nodes
. DetNet Flows
. DetNet Flow Types
. Source Transmission Behavior
. Incomplete Networks
. Traffic Engineering for DetNet
. The Application Plane
. The Controller Plane
. The Network Plane
. Queuing, Shaping, Scheduling, and Preemption
. Service Instance
. Flow Identification at Technology Borders
. Exporting Flow Identification
. Flow Attribute Mapping between Layers
. Flow-ID Mapping Examples
. Advertising Resources, Capabilities, and Adjacencies
. Scaling to Larger Networks
. Compatibility with Layer 2
. Security Considerations
. Privacy Considerations
. IANA Considerations
. Informative References
Acknowledgements
Authors' Addresses
Introduction
This document provides the overall architecture for Deterministic
Networking (DetNet), which provides a capability for the delivery of
data flows with extremely low packet loss rates and bounded end-to-end
delivery latency. DetNet is for networks that are under a single
administrative control or within a closed group of administrative
control; these include campus-wide networks and private WANs. DetNet
is not for large groups of domains such as the Internet.
DetNet operates at the IP layer and delivers service over lower-layer
technologies such as MPLS and IEEE 802.1 Time-Sensitive Networking
(TSN).
DetNet provides a reliable and available service by dedicating network
resources such as link bandwidth and buffer space to DetNet flows and/or
classes of DetNet flows, and by replicating packets along multiple paths.
Unused reserved resources are available to non-DetNet packets as long as all
guarantees are fulfilled. The "Deterministic
Networking Problem Statement" introduces DetNet, and "Deterministic Networking Use Cases" summarizes the
need for it. See for specific techniques
that can be used to identify DetNet flows and assign them to specific paths
through a network. A goal of DetNet is a converged network in all
respects, including the convergence of sensitive non-IP networks onto a common
network infrastructure. The presence of DetNet flows does not preclude
non-DetNet flows, and the benefits offered DetNet flows should not, except in
extreme cases, prevent existing Quality-of-Service (QoS) mechanisms from
operating in a normal fashion, subject to the bandwidth required for the
DetNet flows. A single source-destination pair can trade both DetNet and
non-DetNet flows. End systems and applications need not instantiate special
interfaces for DetNet flows. Networks are not restricted to certain
topologies; connectivity is not restricted. Any application that generates a
data flow that can be usefully characterized as having a maximum bandwidth
should be able to take advantage of DetNet, as long as the necessary resources
can be reserved.
Reservations can be made by the application itself, via network management,
centrally by an application's controller, or by other means, for instance, by
placing on-demand reservation via a distributed Control Plane, e.g.,
leveraging the Resource Reservation Protocol (RSVP) .
QoS requirements of DetNet flows can be met if all network
nodes in a DetNet domain implement DetNet capabilities. DetNet nodes can be
interconnected with different sub-network technologies () where the nodes of the subnet are not DetNet aware
(). Many applications that are intended to be
served by DetNet require the ability to synchronize the clocks in end systems
to a sub-microsecond accuracy. Some of the queue-control techniques defined
in also require time synchronization among
network nodes. The means used to achieve time synchronization are not
addressed in this document. DetNet can accommodate various
time-synchronization techniques and profiles that are defined elsewhere to
address the needs of different market segments.
TerminologyTerms Used in This Document
The following terms are used in the context of DetNet in this document:
allocation
The dedication of resources
to support a DetNet flow. Depending on an
implementation, the resource may be reused by
non-DetNet flows when it is not used by the DetNet
flow.
App-flow
The payload (data) carried over a DetNet service.
DetNet compound flow and DetNet member flow
A DetNet compound
flow is a DetNet flow that has been separated into
multiple duplicate DetNet member flows for service
protection at the DetNet service sub-layer. Member
flows are merged back into a single DetNet compound
flow such that there are no duplicate packets.
"Compound" and "member" are strictly relative to
each other, not absolutes; a DetNet compound flow
comprising multiple DetNet member flows can, in
turn, be a member of a higher-order compound.
DetNet destination
An end system capable of terminating a DetNet flow.
DetNet domain
The portion of a network that
is DetNet aware. It includes end systems and DetNet
nodes.
DetNet edge node
An instance
of a DetNet relay node that acts as a source and/or
destination at the DetNet service sub-layer. For
example, it can include a DetNet service sub-layer
proxy function for DetNet service protection (e.g.,
the addition or removal of packet sequencing
information) for one or more end systems, it can start
or terminate resource allocation at the DetNet
forwarding sub-layer, or it can aggregate DetNet services
into new DetNet flows. It is analogous to a Label
Edge Router (LER) or a Provider Edge (PE) router.
DetNet flow
A sequence of packets that conforms uniquely
to a flow identifier and to which the DetNet service is to be
provided. It includes any DetNet headers added to support the
DetNet service and forwarding sub-layers.
DetNet forwarding sub-layer
DetNet functionality is divided into two sub-layers. One of
them is the DetNet forwarding sub-layer, which optionally
provides resource allocation for DetNet flows over paths
provided by the underlying network.
DetNet intermediate node
A DetNet relay node or DetNet transit node.
DetNet node
A
DetNet edge node, a DetNet relay
node, or a DetNet transit node.
DetNet relay node
A DetNet
node that includes a service
sub-layer function that interconnects different
DetNet forwarding sub-layer paths to provide service
protection. A DetNet relay node participates in the
DetNet service sub-layer. It typically incorporates
DetNet forwarding sub-layer functions as well, in
which case it is collocated with a transit node.
DetNet service sub-layer
DetNet functionality is divided into two sub-layers. One of
them is the DetNet service sub-layer, at which a DetNet
service (e.g., service protection) is provided.
DetNet service proxy
A proxy that maps between App-flows and DetNet flows.
DetNet source
An end system capable of originating a DetNet flow.
DetNet system
A DetNet-aware end system, transit node, or relay node.
"DetNet" may be omitted in some text.
DetNet transit node
A DetNet node, operating at the DetNet forwarding sub-layer,
that utilizes link-layer and/or network-layer
switching across multiple links and/or sub-networks
to provide paths for DetNet service sub-layer
functions. It typically provides resource allocation
over those paths. An MPLS Label Switch Router (LSR) is an example of a
DetNet transit node.
DetNet-UNI
A User-to-Network Interface (UNI) with DetNet-specific
functionalities. It is a packet-based reference
point and may provide multiple functions like
encapsulation, status, synchronization, etc.
end system
Commonly called a "host" in the RFC series and an
"end station" in IEEE 802 standards. End systems of
interest to this document are either sources or
destinations of DetNet flows, and they may or may
not be aware of DetNet forwarding sub-layers or
DetNet service sub-layers.
link
A connection between two DetNet nodes. It may be
composed of a physical link or a sub-network
technology that can provide appropriate traffic
delivery for DetNet flows.
Packet Elimination Function (PEF)
A function that eliminates duplicate
copies of packets to prevent excess packets flooding the
network or duplicate packets being sent out of the DetNet
domain. A PEF can be implemented by a DetNet edge node, a
DetNet relay node, or an end system.
Packet Replication Function (PRF)
A function that replicates DetNet flow
packets and forwards them to one or more next hops in the
DetNet domain. The number of packet copies sent to the
next hops is a parameter specific to the DetNet flow at the point
of replication. A PRF can be implemented by a DetNet edge
node, a DetNet relay node, or an end system.
PREOF
A collective name for Packet Replication, Elimination, and Ordering Functions.
Packet Ordering Function (POF)
A function that reorders packets within
a DetNet flow that are received out of order. This
function can be implemented by a DetNet edge node, a
DetNet relay node, or an end system.
reservation
The set of resources allocated between a source and
one or more destinations through DetNet nodes and
subnets associated with a DetNet flow in order to provide
the provisioned DetNet service.
Dictionary of Terms Used by TSN and DetNet
This section serves as a dictionary for translating the
terms used by the Time-Sensitive Networking (TSN) Task Group
of the IEEE 802.1 WG to those of
the Deterministic Networking (detnet) WG of the IETF.
Listener
The term used by IEEE 802.1 for a destination of a DetNet flow.
Relay system
The term used by IEEE
802.1 for a DetNet intermediate node.
Stream
The term used by IEEE 802.1 for a DetNet flow.
Talker
The term used by IEEE
802.1 for the source of a DetNet flow.
Providing the DetNet Quality of ServicePrimary Goals Defining the DetNet QoS
The DetNet QoS can be expressed in terms of:
Minimum and maximum end-to-end latency from source to
destination, timely delivery, and bounded jitter
(packet delay variation) derived from these
constraints.
Packet loss ratio under various assumptions as to the operational
states of the nodes and links.
An upper bound on out-of-order packet delivery. It is worth noting
that some DetNet applications are unable to tolerate any
out-of-order delivery.
It is a distinction of DetNet that it is concerned solely with
worst-case values for the end-to-end latency, jitter, and
misordering. Average, mean, or typical values are of little
interest, because they do not affect the ability of a real-time
system to perform its tasks. In general, a trivial priority-based
queuing scheme will give better average latency to a data flow than
DetNet; however, it may not be a suitable option for DetNet because
of its worst-case latency.
Three techniques are used by DetNet to provide these qualities of service:
Resource allocation ()
Service protection ()
Explicit routes ()
Resource allocation operates by assigning resources, e.g., buffer
space or link bandwidth, to a DetNet flow (or flow aggregate) along
its path. Resource allocation greatly reduces, or even eliminates
entirely, packet loss due to output packet contention within the
network, but it can only be supplied to a DetNet flow that is limited
at the source to a maximum packet size and transmission rate.
As DetNet flows are assumed to be rate limited and DetNet is designed
to provide sufficient allocated resources (including provisioned
capacity), the use of transport-layer congestion control
for App-flows is not required; however,
if resources are allocated appropriately, use of congestion control
should not impact transmission negatively.
Resource allocation addresses two of the DetNet QoS requirements: latency
and packet loss. Given that DetNet nodes have a finite amount of buffer space,
resource allocation necessarily results in a maximum end-to-end
latency. Resource allocation also addresses contention-related packet loss.
Other important contributions to packet loss are random media errors
and equipment failures. Service protection is the name for the mechanisms
used by DetNet to address these losses. The mechanisms employed are
constrained by the need to meet the users' latency requirements. Packet
replication and elimination () and packet encoding
() are described in this document to provide
service protection, but other mechanisms may also be found. For instance,
packet encoding can be used to provide service protection against random media
errors, while packet replication and elimination can be used to provide
service protection against equipment failures. This mechanism distributes the
contents of DetNet flows over multiple paths in time and/or space, so that the
loss of some of the paths does need not cause the loss of any packets.
The paths are typically (but not necessarily) explicit routes so that
they do not normally suffer temporary interruptions caused by the convergence
of routing or bridging protocols. These three techniques can be applied individually or applied together; it
results that eight combinations, including none (no DetNet), are
possible. Some combinations, however, are of wider utility than others. This
separation keeps the protocol stack coherent and maximizes interoperability
with existing and developing standards in the IETF and other Standards
Development Organizations. The following are examples of typical expected
combinations:
The combination of explicit routes and service protection is the technique
employed by seamless redundancy mechanisms applied on a ring topology,
e.g., as described in . In this
example, explicit routes are achieved by limiting the physical
topology of the network to a ring. Sequentialization, replication, and
duplicate elimination are facilitated by packet tags added at the
front or the end of Ethernet frames. provides
another example in the context of MPLS.
Resource allocation
alone was originally offered by Audio Video Bridging as defined by IEEE 802.1 . As long as the network suffers no failures,
packet loss due to output packet contention can be eliminated through
the use of a reservation protocol (e.g., the Multiple Stream Registration
Protocol ), shapers in every bridge,
and proper dimensioning.
Using all three together gives
maximum protection.
There are, of course, simpler methods available (and employed today) to
achieve levels of latency and packet loss that are satisfactory for many
applications. Prioritization and over-provisioning is one such technique.
However, these methods generally work best in the absence of any significant
amount of noncritical traffic in the network (if, indeed, such traffic is
supported at all). They may also work only if the critical traffic constitutes only a small portion of
the network's theoretical capacity, if all systems are functioning properly,
or if actions by end systems that disrupt the network's
operations are absent. There are any number of methods in use, defined, or in progress for
accomplishing each of the above techniques. It is expected that the DetNet
architecture defined in this document will assist various vendors, users, and/or "vertical" Standards
Development Organizations (dedicated to a single industry) in making selections
among the available means of implementing DetNet networks.
Mechanisms to Achieve DetNet QoSResource AllocationEliminate Contention Loss
The primary means by which DetNet achieves its QoS
assurances is to reduce, or even completely eliminate,
packet loss due to output packet contention within a
DetNet node as a cause of packet loss. This can be
achieved only by the provision of sufficient buffer
storage at each node through the network to ensure
that no packets are dropped due to a lack of buffer
storage. Note that App-flows are generally not
expected to be responsive to implicit or explicit congestion notification
.
Ensuring adequate buffering requires, in turn, that
the source and every DetNet node along the path to the
destination (or nearly every node; see ) be careful to regulate its output to
not exceed the data rate for any DetNet flow, except for brief
periods when making up for interfering traffic. Any packet
sent ahead of its time potentially adds to the number of
buffers required by the next-hop DetNet node and may thus
exceed the resources allocated for a particular DetNet
flow. Furthermore, rate limiting (e.g., using traffic policing)
and shaping functions (e.g., shaping as defined in ) at the
ingress of the DetNet domain must be applied. This is needed
for meeting the requirements of DetNet flows as well as for
protecting non-DetNet traffic from potentially misbehaving
DetNet traffic sources. Note that large buffers have some
issues (see, e.g., ). The low-level mechanisms described in
provide the necessary regulation of transmissions by an end system or DetNet
node to provide resource allocation. The allocation of the bandwidth and
buffers for a DetNet flow requires provisioning. A DetNet node may have other
resources requiring allocation and/or scheduling that might otherwise be
over-subscribed and trigger the rejection of a reservation.
Jitter Reduction
A core objective of DetNet is to enable the convergence of sensitive non-IP networks
onto a common network infrastructure. This requires the accurate emulation
of currently deployed mission-specific networks, which,
for example, rely on point-to-point analog (e.g., 4-20mA modulation) and
serial-digital cables (or buses) for highly reliable, synchronized, and
jitter-free communications. While the latency of analog transmissions is
basically the speed of light, legacy serial links are usually slow (in the
order of Kbps) compared to, say, Gigabit Ethernet, and some latency is usually
acceptable. What is not acceptable is the introduction of excessive jitter,
which may, for instance, affect the stability of control systems.
Applications that are designed to operate on serial links usually do
not provide services to recover the jitter, because jitter simply does not
exist there. DetNet flows are generally expected to be delivered in order,
and the precise time of reception influences the processes. In order to
converge such existing applications,
there is a desire to emulate all properties of the serial cable, such
as clock transportation, perfect flow isolation, and fixed latency. While minimal
jitter (in the form of specifying minimum, as well as maximum, end-to-end latency)
is supported by DetNet, there are practical limitations on packet-based networks
in this regard. In general, users
are encouraged to use a combination of:
Sub-microsecond time synchronization among all source and destination
end systems, and
Time-of-execution fields in the application packets.
Jitter reduction is provided by the mechanisms described in
that also provide resource allocation.
Service Protection
Service protection aims to mitigate or eliminate packet loss due
to equipment failures, including random media and/or memory
faults. These types of packet loss can be greatly reduced by
spreading the data over multiple disjoint forwarding
paths. Various service protection methods are described in , e.g., 1+1 linear protection. The functional
details of an additional method are described in , which can be implemented as described in
or as specified in in order to provide 1+n hitless
protection. The appropriate service protection mechanism depends
on the scenario and the requirements.
In-Order Delivery
Out-of-order packet delivery can be a side effect of service
protection. Packets delivered out of order impact the amount of
buffering needed at the destination to properly process the received
data. Such packets also influence the jitter of a flow. The guarantees
of a DetNet service include a maximum amount of misordering as a
constraint. Zero misordering would be a valid service constraint to
reflect that the end system(s) of the flow cannot tolerate any
out-of-order delivery. A DetNet Packet Ordering Function (POF)
() can be used to provide in-order delivery.
Packet Replication and Elimination
This section describes a service protection method that sends copies
of the same packets over multiple paths. The DetNet service
sub-layer includes the PRF, PEF, and POF for use in DetNet edge,
relay node, and end-system packet processing. These functions can be
enabled in a DetNet edge node, relay node, or end system. The
collective name for all three functions is Packet Replication,
Elimination, and Ordering Functions (PREOF). The packet replication
and elimination service protection method altogether involves four
capabilities:
Sequencing information is provided to the
packets of a DetNet compound flow. This may
be done by adding a sequence number or time
stamp as part of DetNet, or it may be inherent
in the packet, e.g., in a higher-layer
protocol or associated to other physical
properties such as the precise time (and radio
channel) of reception of the packet. This is
typically done once, at or near the source.
The PRF
replicates these packets into multiple DetNet
member flows and typically sends them along
multiple different paths to the
destination(s), e.g., over the explicit routes
described in . The location
within a DetNet node and the mechanism used
for the PRF are left open for implementations.
The PEF
eliminates duplicate packets of a DetNet flow
based on the sequencing information and a
history of received packets. The output of the
PEF is always a single packet. This may be
done at any DetNet node along the path to save
network resources further downstream, in
particular if multiple replication points
exist. But the most common case is to perform
this operation at the very edge of the DetNet
network, preferably in or near the
receiver. The location within a DetNet node
and the mechanism used for the PEF is left open
for implementations.
The
POF uses the sequencing
information to reorder a DetNet flow's packets
that are received out of order.
The order in which a DetNet node applies PEF, POF, and
PRF to a DetNet flow is left open for implementations.
Some service protection mechanisms rely on switching from one flow to
another when a failure of a flow is detected. Contrarily, packet replication
and elimination combines the DetNet member flows sent along multiple different
paths and performs a packet-by-packet selection of which to discard, e.g.,
based on sequencing information.
In the simplest case, this amounts to 1) replicating each packet in a
source that has two interfaces and 2) conveying them through the network along
separate (Shared Risk Link Group (SRLG) disjoint) paths to the similarly
dual-homed destinations that 3) reorder the packets and 4) discard the
duplicates. This ensures that one path
remains, even if some DetNet intermediate node fails. The sequencing
information can also be used for loss detection and for reordering.
DetNet relay nodes in the network can provide replication and elimination
facilities at various points in the network so that multiple failures can be
accommodated. This is shown in , where the two relay nodes
each replicate (R) the DetNet flow on input, sending the DetNet member flows
to both the other relay node and to the end system, and eliminate duplicates
(E) on the output interface to the right-hand end system. Any one link in the
network can fail, and the DetNet compound flow can still get through.
Furthermore, two links can fail, as long as they are in different segments of
the network.
Packet replication and elimination does not react to and correct failures;
it is entirely passive. Thus, intermittent failures, mistakenly created
packet filters, or misrouted data is handled just the same as the equipment
failures that are handled by typical routing and bridging protocols. If member flows that take different-length paths through the network are
combined, a merge point may require extra buffering to equalize the delays
over the different paths. This equalization ensures that the resultant
compound flow will not exceed its contracted bandwidth even after one of the
paths is restored after a failure. The extra buffering can be also used to
provide in-order delivery.
Packet Encoding for Service Protection
There are methods for using multiple paths to provide service protection
that involve encoding the information in a
packet belonging to a DetNet flow into multiple transmission units,
combining information from multiple packets into any given transmission unit.
Such techniques, also known as "network coding",
can be used as a DetNet service protection technique.
Explicit Routes
In networks controlled by typical dynamic control protocols such as IS-IS or
OSPF, a network topology event in one part of the network can impact, at least
briefly, the delivery of data in parts of the network remote from the failure
or recovery event. Even the use of redundant paths through a network, e.g., as
defined by , does not eliminate the chances of packet
loss. Furthermore, out-of-order packet delivery can be a side effect of route
changes. Many real-time networks rely on physical rings of two-port
devices, with a relatively simple ring control protocol. This supports
redundant paths for service protection with a minimum of wiring. As an
additional benefit, ring topologies can often utilize different topology
management protocols from those used for a mesh network, with a consequent
reduction in the response time to topology changes. Of course, this comes at
some cost in terms of increased hop count, and thus latency, for the typical
path. In order to get the advantages of low hop count and still ensure against
even very brief losses of connectivity, DetNet employs explicit routes where
the path taken by a given DetNet flow does not change, at least not
immediately and likely not at all, in response to network topology events.
Service protection (see Sections
and ) over explicit routes
provides a high likelihood of continuous connectivity. Explicit routes can be
established in various ways, e.g., with RSVP-TE , with
Segment Routing (SR) , via a SDN approach , with IS-IS , etc. Explicit routes
are typically used in MPLS TE (Traffic Engineering) Label Switched Paths
(LSPs). Out-of-order packet delivery can be a side effect of distributing a single
flow over multiple paths, especially when there is a change from one path to
another when combining the flow. This is irrespective of the distribution
method used and also applies to service protection over explicit routes. As
described in , out-of-order packets influence the
jitter of a flow and impact the amount of buffering needed to process the
data; therefore, the guarantees of a DetNet service include a maximum amount
of misordering as a constraint. The use of explicit routes helps to provide in-order delivery
because there is no immediate route change with the network topology, but the
changes are plannable as they are between the different explicit routes.
Secondary Goals for DetNet
Many applications require DetNet to provide additional services, including coexistence with
other QoS mechanisms () and protection against misbehaving transmitters
().
Coexistence with Normal Traffic
A DetNet network supports the dedication of a high proportion of
the network bandwidth to DetNet flows. But, no matter how much
is dedicated for DetNet flows, it is a goal of DetNet to coexist
with existing Class-of-Service schemes (e.g., DiffServ). It is
also important that non-DetNet traffic not disrupt the DetNet
flow, of course (see Sections and ). For these reasons:
Bandwidth (transmission opportunities) not utilized by a DetNet flow is
available to non-DetNet packets (though not to other DetNet flows).
DetNet flows can be shaped or scheduled, in order to ensure that the
highest-priority non-DetNet packet is also ensured a worst-case latency.
When transmission opportunities for DetNet flows are scheduled in detail,
the algorithm constructing the schedule should leave sufficient
opportunities for non-DetNet packets to satisfy the needs of the users of the
network. Detailed scheduling can also permit the time-shared use of buffer
resources by different DetNet flows.
Starvation of non-DetNet traffic must be avoided, for example, by
traffic policing and shaping functions (e.g., ). Thus, the net effect of the presence of DetNet
flows in a network on the non-DetNet flows is primarily a reduction
in the available bandwidth.
Fault Mitigation
Robust real-time systems require reducing the number of possible
failures. Filters and policers should be used in a DetNet
network to detect if DetNet packets are received on the wrong
interface, at the wrong time, or in too great a volume.
Furthermore, filters and policers can take actions to discard
the offending packets or flows, or trigger shutting down the
offending flow or the offending interface. It is also
essential that filters and service remarking be employed at the
network edge to prevent non-DetNet packets from being mistaken
for DetNet packets and thus impinging on the resources
allocated to DetNet packets. In particular, sending DetNet
traffic into networks that have not been provisioned in advance
to handle that DetNet traffic has to be treated as a fault. The
use of egress traffic filters, or equivalent mechanisms, to
prevent this from happening are strongly recommended at the
edges of DetNet networks and DetNet supporting networks. In
this context, the term 'provisioned' has a broad meaning, e.g.,
provisioning could be performed via an administrative decision
that the downstream network has the available capacity to carry
the DetNet traffic that is being sent into it.
Note that the sending of App-flows that do not use
transport-layer congestion control per into a network that is not
provisioned to handle such traffic has to be treated
as a fault and prevented. PRF-generated DetNet
member flows also need to be treated as not using
transport-layer congestion control even if the
original App-flow supports transport-layer
congestion control because PREOF can remove
congestion indications at the PEF and thereby hide
such indications (e.g., drops, ECN markings,
increased latency) from end systems.
The mechanisms to support these requirements are both Data
Plane and implementation specific. Solutions that are data-plane
specific will be specified in the relevant data-plane solution
document. There also exist techniques, at present and/or in various
stages of standardization, that can support these fault-mitigation
tasks that deliver a high probability that misbehaving systems will
have zero impact on well-behaved DetNet flows with the exception, of
course, of the receiving interface(s) immediately downstream from
the misbehaving device. Examples of such techniques include traffic
policing and shaping functions (e.g., those described in ),
separating flows into per-flow rate-limited queues, and potentially
applying active queue management .
DetNet ArchitectureDetNet Stack Model
DetNet functionality () is implemented
in two adjacent sub-layers in the protocol stack: the DetNet service
sub-layer and the DetNet forwarding sub-layer. The DetNet service sub-layer
provides DetNet service, e.g., service protection, to higher layers
in the protocol stack and applications. The DetNet forwarding sub-layer
supports DetNet service in the underlying network, e.g., by
providing explicit routes and resource allocation to DetNet flows.
Representative Protocol Stack Model illustrates a conceptual DetNet data-plane layering model.
One may compare it to that in , Annex C.
Not all sub-layers are required for any given application, or even for any
given network. The functionality shown in is:
Application
Shown as "source" and "destination" in the diagram.
Packet sequencing
As part of the DetNet service sub-layer, the packet
sequencing function supplies the sequence number for
packet replication and elimination for DetNet service
protection (); thus, its peer is
duplicate elimination. This sub-layer is not needed if a
higher-layer protocol is expected to perform any packet
sequencing and duplicate elimination required by the
DetNet flow replication.
Duplicate elimination
As part of the DetNet service sub-layer, based on the sequence number
supplied by its peer (packet sequencing),
duplicate elimination discards any duplicate packets generated by DetNet
flow replication. It can operate on member flows, compound flows, or both.
The replication may also be inferred from other
information such as the precise time of reception in a scheduled network.
The duplicate elimination sub-layer may also perform resequencing of packets
to restore packet order in a
flow that was disrupted by the loss of packets on one or another of
the multiple paths taken.
Flow replication
As
part of DetNet service protection, packets that belong to
a DetNet compound flow are replicated into two or more
DetNet member flows. This function is separate from
packet sequencing. Flow replication can be an explicit
replication and remarking of packets or can be performed
by, for example, techniques similar to ordinary multicast
replication, albeit with resource allocation implications.
Its peer is DetNet flow merging.
Flow merging
As
part of the DetNet service sub-layer, the flow merging
function combines DetNet member flows together for packets
coming up the stack belonging to a specific DetNet
compound flow. DetNet flow merging, together with packet
sequencing, duplicate elimination, and DetNet flow
replication perform packet replication and elimination
(). Its peer is DetNet flow
replication.
Packet encoding
As
part of DetNet service protection, as an alternative to
packet sequencing and flow replication, packet encoding
combines the information in multiple DetNet packets,
perhaps from different DetNet compound flows, and
transmits that information in packets on different DetNet
member flows. Its peer is packet decoding.
Packet decoding
As
part of DetNet service protection, as an alternative to
flow merging and duplicate elimination, packet decoding
takes packets from different DetNet member flows and
computes from those packets the original DetNet packets
from the compound flows input to packet encoding. Its
peer is packet encoding.
Resource allocation
The DetNet forwarding sub-layer provides resource
allocation. See . The
actual queuing and shaping mechanisms are typically
provided by the underlying subnet. These can be closely
associated with the means of providing paths for DetNet
flows. The path and the resource allocation are conflated
in this figure.
Explicit routes
Explicit routes are arrangements of fixed paths operated at
the DetNet forwarding sub-layer that are determined in advance
to avoid the impact of network convergence on DetNet flows.
Operations, Administration, and Maintenance (OAM) leverages
in-band and out-of-band signaling that validates whether the
service is effectively obtained within QoS constraints. OAM is
not shown in ; it may reside in any
number of the layers. OAM can involve specific tagging added in
the packets for tracing implementation or network configuration
errors; traceability enables finding whether a packet is a
replica, which DetNet relay node performed the replication, and
which segment was intended for the replica. Active and hybrid OAM
methods require additional bandwidth to perform fault management
and performance monitoring of the DetNet domain. OAM may, for
instance, generate special test probes or add OAM information into
the data packet.
The packet replication and elimination functions may be
performed either at the source and destination ends of a
DetNet compound flow or in a DetNet relay node.
DetNet Data-Plane Overview
A "Deterministic Network" will be composed of DetNet-enabled end
systems, DetNet edge nodes, and DetNet relay nodes, which
collectively deliver DetNet services. DetNet relay and edge nodes
are interconnected via DetNet transit nodes (e.g., LSRs), which
support DetNet but are not DetNet service aware. All DetNet nodes
are connected to sub-networks, where a point-to-point link is also
considered a simple sub-network. These sub-networks provide
DetNet-compatible service for support of DetNet traffic. Examples
of sub-network technologies include MPLS TE, TSN as defined by
IEEE 802.1, and OTN (Optical Transport Network). Of course,
multilayer DetNet systems may also be possible, where one DetNet
appears as a sub-network and provides service to a higher-layer
DetNet system. A simple DetNet concept network is shown in .
Note that in this and following figures, "Forwarding" and "Fwd" refer to the
DetNet forwarding sub-layer, and "Service" and "Svc" refer to the DetNet
service sub-layer; both of these sub-layers are described in detail in .
DetNet Data Plane is divided into two sub-layers: the DetNet
service sub-layer and the DetNet forwarding sub-layer. This helps
to explore and evaluate various combinations of the data-plane
solutions available. Some of them are illustrated in . This separation of DetNet sub-layers,
while helpful, should not be considered a formal requirement.
For example, some technologies may violate these strict sub-layers
and still be able to deliver a DetNet service.
In some networking scenarios, the end system initially provides a
DetNet flow encapsulation, which contains all information needed
by DetNet nodes (e.g., DetNet flow based on the Real-time Transport
Protocol (RTP) that is carried over a
native UDP/IP network or pseudowire (PW)). In other scenarios, the
encapsulation formats might differ significantly.
There are many valid options to create a data-plane solution for DetNet
traffic by selecting a technology approach for the DetNet service sub-layer and
also selecting a technology approach for the DetNet forwarding sub-layer. There are
a large number of valid combinations.
One of the most fundamental differences between different
potential data-plane options is the basic headers used by DetNet
nodes. For example, the basic service can be delivered based on
an MPLS label or an IP header. This decision impacts the basic
forwarding logic for the DetNet service sub-layer. Note that in
both cases, IP addresses are used to address DetNet nodes. The
selected DetNet forwarding sub-layer technology also needs to be
mapped to the subnet technology used to interconnect DetNet
nodes. For example, DetNet flows will need to be mapped to TSN
Streams.
Network Reference Model shows another view of the
DetNet service-related reference points and main components.
DetNet User-to-Network Interfaces (DetNet-UNIs) ("U" in ) are assumed in this document to be
packet-based reference points and provide connectivity over the
packet network. A DetNet-UNI may provide multiple functions. For
example, it may:
add encapsulation specific to networking technology to the DetNet flows if necessary,
provide status of the availability of the resources associated with a reservation,
provide a synchronization service for the end system, or
carry enough signaling to place the reservation in a network without a
controller or in a network where the controller only deals with the network
but not the end systems.
Internal reference points of end systems (between the application
and the Network Interface Card (NIC)) are more challenging from the
control perspective, and they may have extra requirements (e.g.,
in-order delivery is expected in end system internal reference
points, whereas it is considered optional over the DetNet-UNI).DetNet SystemsEnd System
The traffic characteristics of an App-flow can be CBR
(constant bit rate) or VBR (variable bit rate) and can have
Layer 1, Layer 2, or Layer 3 encapsulation (e.g., TDM
(time-division multiplexing) Ethernet, IP). These
characteristics are considered as input for resource
reservation and might be simplified to ensure determinism
during packet forwarding (e.g., making reservations for the
peak rate of VBR traffic, etc.).
An end system may or may not be aware of the DetNet forwarding sub-layer
or DetNet service sub-layer. That is, an end
system may or may not contain DetNet-specific
functionality. End systems with DetNet functionalities may
have the same or different forwarding sub-layer as the
connected DetNet domain. Categorization of end systems are
shown in .
The following are some known use case examples for end systems:
DetNet unaware
The classic case requiring service proxies.
DetNet f-aware
A system that is aware of the DetNet forwarding
sub-layer. It knows about some TSN
functions (e.g., reservation) but not about service
protection.
DetNet s-aware
A system that is aware of the DetNet service
sub-layer. It supplies sequence numbers but
doesn't know about resource allocation.
DetNet sf-aware
A fully functioning DetNet end
system. It has DetNet functionalities and usually the
same forwarding paradigm as the connected DetNet
domain. It can be treated as an integral part of the
DetNet domain.
DetNet Edge, Relay, and Transit Nodes
As shown in , DetNet edge nodes
providing proxy service and DetNet relay nodes providing the
DetNet service sub-layer are DetNet aware, and DetNet transit
nodes need only be aware of the DetNet forwarding sub-layer.
In general, if a DetNet flow passes through one or more
DetNet-unaware network nodes between two DetNet nodes providing
the DetNet forwarding sub-layer for that flow, there is a
potential for disruption or failure of the DetNet QoS. A network
administrator needs to 1) ensure that the DetNet-unaware network
nodes are configured to minimize the chances of packet loss and
delay and 2) provision enough extra buffer space in the DetNet
transit node following the DetNet-unaware network nodes to absorb
the induced latency variations.
DetNet FlowsDetNet Flow Types
A DetNet flow can have different formats while its packets are forwarded
between the peer end systems depending on the type of the end
systems. Corresponding to the end system types, the following possible
types/formats of a DetNet flow are distinguished in this document. The
different flow types have different requirements to DetNet nodes.
App-flow
The payload (data) carried over a DetNet
flow between DetNet-unaware end systems. An App-flow does
not contain any DetNet-related attributes and does not
imply any specific requirement on DetNet nodes.
DetNet-f-flow
The specific format of a DetNet flow. It
only requires the resource allocation features provided by
the DetNet forwarding sub-layer.
DetNet-s-flow
The specific format of a DetNet flow. It
only requires the service protection feature ensured by
the DetNet service sub-layer.
DetNet-sf-flow
The specific format of a DetNet
flow. It requires both the DetNet service sub-layer and
the DetNet forwarding sub-layer functions during
forwarding.
Source Transmission Behavior
For the purposes of resource allocation,
DetNet flows can be synchronous or asynchronous.
In synchronous DetNet flows, at least the DetNet nodes (and possibly
the end systems) are closely time
synchronized, typically to better than 1 microsecond. By transmitting
packets from different DetNet flows or classes of DetNet flows at different times,
using repeating schedules synchronized among the DetNet nodes, resources
such as buffers and link bandwidth can be shared over the time domain
among different DetNet flows. There is a trade-off among techniques for
synchronous DetNet flows between the burden of fine-grained scheduling and the
benefit of reducing the required resources, especially buffer space.
In contrast, asynchronous DetNet flows are not coordinated with a fine-grained
schedule, so relay and end systems must assume worst-case interference
among DetNet flows contending for buffer resources.
Asynchronous DetNet flows are characterized by:
A maximum packet size;
An observation interval; and
A maximum number of transmissions during that observation interval.
These parameters, together with knowledge of the
protocol stack used (and thus the size of the various headers
added to a packet), provide the bandwidth that is needed for the
DetNet flow. The source is required not to exceed these
limits in order to obtain DetNet service. If the source
transmits less data than this limit allows, then the unused resource,
such as link bandwidth, can be made available by the DetNet
system to non-DetNet packets as long as all guarantees are
fulfilled. However, making those resources available to DetNet
packets in other DetNet flows would serve no purpose. Those
other DetNet flows have their own dedicated resources, on the
assumption that all DetNet flows can use all of their resources
over a long period of time.
There is no expectation in DetNet for App-flows to be responsive to
congestion control or explicit congestion
notification . The assumption is that a DetNet
flow, to be useful, must be delivered in its entirety. That is, while
any useful application is written to expect a certain number of lost
packets, the real-time applications of interest to DetNet demand that the
loss of data due to the network is a rare event. Although DetNet strives to minimize the changes required of an
application to allow it to shift from a special-purpose digital network to an
Internet Protocol network, one fundamental shift in the behavior of network
applications is impossible to avoid: the reservation of resources before the
application starts. In the first place, a network cannot deliver finite
latency and practically zero packet loss to an arbitrarily high offered load.
Secondly, achieving practically zero packet loss for DetNet flows means that
DetNet nodes have to dedicate buffer resources to specific DetNet flows or to
classes of DetNet flows. The requirements of each reservation have to be
translated into the parameters that control each DetNet system's queuing,
shaping, and scheduling functions, and they have to be delivered to the DetNet nodes and end
systems. All nodes in a DetNet domain are expected to support the data behavior
required to deliver a particular DetNet service. If a node itself is not
DetNet service aware, the DetNet nodes that are adjacent to them must ensure
that the node that is non-DetNet aware is provisioned to appropriately support
the DetNet service. For example, a TSN node (as defined by IEEE 802.1) may be used to
interconnect DetNet-aware nodes, and these DetNet nodes can map DetNet flows
to 802.1 TSN flows. As another example, an MPLS-TE or MPLS-TP (Transport Profile) domain may be used to
interconnect DetNet-aware nodes, and these DetNet nodes can map DetNet flows
to TE LSPs, which can provide the QoS requirements of the DetNet service.
Incomplete Networks
The presence in the network of intermediate nodes or subnets
that are not fully capable of offering DetNet services
complicates the ability of the intermediate nodes and/or
controller to allocate resources, as extra buffering must be
allocated at points downstream from the non-DetNet
intermediate node for a DetNet flow. This extra buffering
may increase latency and/or jitter.
Traffic Engineering for DetNetTraffic Engineering Architecture and Signaling (TEAS)
defines traffic-engineering architectures for generic applicability
across packet and nonpacket networks.
From a TEAS perspective, Traffic Engineering (TE) refers to techniques
that enable operators to control how specific traffic flows are treated
within their networks.
Because of its very nature of establishing explicit optimized paths,
DetNet can be seen as a new, specialized branch of
TE, and it inherits its architecture with a separation
into planes.
The DetNet architecture is thus composed of three planes: a (User)
Application Plane, a Controller Plane, and a Network Plane. This
echoes the composition of Figure 1 of "Software-Defined Networking (SDN): Layers and
Architecture Terminology" and the controllers identified in
and .
The Application Plane
Per ,
the Application Plane includes both applications and services. In particular,
the Application Plane incorporates the User Agent, a specialized application
that interacts with the end user and operator and performs requests for
DetNet services via an abstract Flow Management Entity
(FME), which may or may not be collocated with (one of) the end systems.
At the Application Plane, a management interface enables
the negotiation of flows between end systems. An abstraction
of the flow called a Traffic Specification (TSpec) provides
the representation. This abstraction is used to place a
reservation over the (Northbound) Service Interface and within
the Application Plane. It is associated with an abstraction
of location, such as IP addresses and DNS names, to identify
the end systems and possibly specify DetNet nodes.
The Controller Plane
The Controller Plane corresponds to the aggregation of the Control
and Management Planes in , though Common
Control and Measurement Plane (CCAMP) (as defined by the CCAMP
Working Group ) makes an
additional distinction between management and measurement. When the
logical separation of the Control, Measurement, and other Management
entities is not relevant, the term "Controller Plane" is used for
simplicity to represent them all, and the term "Controller Plane
Function (CPF)" refers to any device operating in that plane, whether
it is a Path Computation Element (PCE) , a
Network Management Entity (NME), or a distributed control protocol.
The CPF is a core element of a controller, in charge of computing
deterministic paths to be applied in the Network Plane.
A (Northbound) Service Interface enables applications in the Application
Plane to communicate with the entities in the Controller Plane as
illustrated in .
One or more CPFs collaborate to implement the requests from the FME
as per-flow, per-hop behaviors installed in the DetNet nodes for each
individual flow. The CPFs place each flow along a deterministic
arrangement of DetNet nodes so as to respect per-flow constraints such
as security and latency, and to optimize the overall result for metrics
such as an abstract aggregated cost. The deterministic arrangement can
typically be more complex than a direct arrangement and include
redundant paths with one or more packet replication and elimination
points. Scaling to larger networks is discussed in .
The Network Plane
The Network Plane represents the network devices and protocols as a whole,
regardless of the layer at which the network devices operate. It includes the
Data Plane and Operational Plane (e.g., OAM) aspects.
The Network Plane comprises the Network Interface Cards (NICs) in the
end systems, which are typically IP hosts, and DetNet nodes, which
are typically IP routers and MPLS switches.
A Southbound (Network) Interface enables the entities in the Controller
Plane to communicate with devices in the Network Plane as illustrated
in . This interface leverages and extends
TEAS to describe the physical topology and resources in the Network
Plane.
The DetNet nodes (and possibly the end systems' NICs) expose their capabilities
and physical resources to the controller (the CPF) and update the CPFs with
their dynamic perception of the topology across the Southbound Interface. In
return, the CPFs set the per-flow paths up, providing a Flow Characterization
that is more tightly coupled to the DetNet node operation than a TSpec.
At the Network Plane, DetNet nodes may exchange information regarding
the state of the paths, between adjacent DetNet nodes and possibly with the
end systems, and forward packets within constraints associated to each flow,
or, when unable to do so, perform a last-resort operation such as drop or
declassify. This document focuses on the Southbound interface and the
operation of the Network Plane.
Queuing, Shaping, Scheduling, and Preemption DetNet achieves bounded delivery latency by reserving bandwidth and buffer
resources at each DetNet node along the path of the DetNet flow. The
reservation itself is not sufficient, however. Implementors and users of a
number of proprietary and standard real-time networks have found that
standards for specific data-plane techniques are required to enable these
assurances to be made in a multivendor network. The fundamental reason is
that latency variation in one DetNet system results in the need for extra
buffer space in the next-hop DetNet system(s), which in turn increases the
worst-case per-hop latency. Standard queuing and transmission-selection algorithms allow TE
() to compute the latency contribution of each
DetNet node to the end-to-end latency, to compute the amount of buffer space
required in each DetNet node for each incremental DetNet flow, and most
importantly, to translate from a flow specification to a set of values for the
managed objects that control each relay or end system. For example, the IEEE
802.1 WG has specified (and is specifying) a set of queuing, shaping, and
scheduling algorithms that enable each DetNet node, and/or a central
controller, to compute these values. These algorithms include:
A credit-based shaper (incorporated to ).
Time-gated queues governed by a rotating time schedule based on
synchronized time (incorporated to ).
Synchronized double (or triple) buffers driven by synchronized time ticks.
(incorporated to ).
Preemption of an Ethernet packet in transmission by a packet with a more
stringent latency requirement, followed by the resumption of the preempted
packet (incorporated to ) (incorporated to ).
While these techniques are currently embedded in
Ethernet and bridging
standards, we can note that they are all, except perhaps for
packet preemption, equally applicable to media other than
Ethernet and to routers as well as bridges. Other media may
have their own methods (see, e.g., and ).
Further techniques are defined by the IETF (e.g., and ). DetNet may
include such definitions in the future or may define how
these techniques can be used by DetNet nodes.
Service Instance A service instance represents all the functions required
on a DetNet node to allow the end-to-end service between the
UNIs.
The DetNet network general reference model is shown in for a DetNet service scenario (i.e., between two
DetNet-UNIs). In this figure, end systems ("A" and "B") are connected directly
to the edge nodes of an IP/MPLS network ("PE1" and "PE2"). End systems
participating in DetNet communication may require connectivity before setting
up an App-flow that requires the DetNet service. Such a connectivity-related
service instance and the one dedicated for DetNet service share the same
access. Packets belonging to a DetNet flow are selected by a filter configured
on the access ("F1" and "F2"). As a result, data-flow-specific access
("access-A + F1" and "access-B + F2") is terminated in the flow-specific
service instance ("SI-1" and "SI-2"). A tunnel is used to provide connectivity
between the service instances.
The tunnel is exclusively used for the packets of the DetNet flow between "SI-1" and "SI-2". The service instances are configured to implement DetNet functions and a flow-specific DetNet forwarding. The service instance and the tunnel may or may not be shared by multiple DetNet flows. Sharing the service instance by multiple DetNet flows requires properly populated forwarding tables of the service instance.
The tunnel between the service instances may have some special
characteristics. For example, in case of a DetNet L3 service, there are
differences in the usage of the PW for DetNet traffic compared to the network
model described in . In the DetNet scenario, the PW is
likely to be used exclusively by the DetNet flow, whereas states:
The packet PW appears as a single point-to-point link to the client
layer. Network-layer adjacency formation and maintenance between the
client equipments will follow the normal practice needed to support
the required relationship in the client layer.
and
This packet pseudowire is used to transport all of the required layer 2 and
layer 3 protocols between LSR1 and LSR2.
Further details are network technology specific and can be found in .
Flow Identification at Technology BordersThis section discusses what needs to be done at technology
borders including Ethernet as one of the technologies. Flow
identification for MPLS and IP Data Planes are described in and ,
respectively.
Exporting Flow Identification
A DetNet node may need to map specific flows to lower-layer
flows (or Streams) in order to provide specific queuing and
shaping services for specific flows. For example:
A non-IP, strictly L2 source end system X may be sending multiple flows
to the same L2 destination end system Y. Those flows may include
DetNet flows with different QoS requirements and may include non-DetNet
flows.
A router may be sending any number of flows to another router.
Again, those flows may include
DetNet flows with different QoS requirements and may include non-DetNet
flows.
Two routers may be separated by bridges. For these bridges to perform
any required per-flow queuing and shaping, they must be able to identify
the individual flows.
A Label Edge Router (LER) may have a Label Switched
Path (LSP) set up for handling traffic
destined for a particular IP address carrying only non-DetNet flows. If
a DetNet flow to that same address is requested, a separate LSP may be
needed in order for all of the Label Switch Routers (LSRs) along the
path to the destination to give that flow special queuing and shaping.
The need for a lower-layer node to be aware of
individual higher-layer flows is not unique to DetNet. But,
given the endless complexity of layering and relayering over
tunnels that is available to network designers, DetNet needs
to provide a model for flow identification that is better than
packet inspection. That is not to say that packet inspection
to Layer 4 or Layer 5 addresses will not be used or the
capability standardized; however, there are alternatives.
A DetNet relay node can connect DetNet flows on different
paths using different flow identification methods. For
example:
A single unicast DetNet flow passing from router A through a bridged network
to router B may be assigned a TSN Stream identifier that is
unique within that bridged network. The bridges can then identify the
flow without accessing higher-layer headers. Of course, the receiving router
must recognize and accept that TSN Stream.
A DetNet flow passing from LSR A to LSR B may be assigned a different
label than that used for other flows to the same IP destination.
In any of the above cases, it is possible that an existing DetNet flow can
be an aggregate carrying multiple other DetNet flows (not to be confused
with DetNet compound vs. member flows). Of course, this requires that the
aggregate DetNet flow be provisioned properly to carry the aggregated flows.
Thus, rather than packet inspection, there is the option to export
higher-layer information to the lower layer. The requirement to support
one or the other method for flow identification (or both) is a
complexity that is part of DetNet control models.
Flow Attribute Mapping between LayersForwarding of packets of DetNet flows over multiple
technology domains may require that lower layers are aware of
specific flows of higher layers. Such an "exporting of flow
identification" is needed each time when the forwarding
paradigm is changed on the forwarding path (e.g., two LSRs are
interconnected by an L2 bridged domain, etc.). The three
representative forwarding methods considered for DetNet
are:
IP routing
MPLS label switching
Ethernet bridging
A packet with corresponding Flow-IDs is illustrated in ,
which also indicates where each Flow-ID can be added or removed.
The additional (domain-specific) Flow-ID can be:
created by a domain-specific function or
derived from the Flow-ID added to the App-flow.
The Flow-ID must be unique inside a given domain. Note that the Flow-ID added to the App-flow is still present in the packet, but some nodes may lack the function to recognize it; that's why the additional Flow-ID is added.
Flow-ID Mapping Examples IP nodes and MPLS nodes are assumed to be configured to push such an additional (domain-specific) Flow-ID when sending traffic to an Ethernet switch (as shown in the examples below).
shows a scenario where an IP end system ("IP-A") is connected via two Ethernet switches ("ETH-n") to an IP router ("IP-1").
End system "IP-A" uses the original App-flow-specific ID
("L3-ID"), but as it is connected to an Ethernet
domain, it has to push an Ethernet-domain-specific
Flow-ID ("ETH-ID") before sending the packet to
"ETH-1". Ethernet switch "ETH-1" can recognize the data flow
based on the "ETH-ID", and it does forwarding toward
"ETH-2". "ETH-2" switches the packet toward the IP
router. "IP-1" must be configured to receive the Ethernet
Flow-ID-specific multicast
flow, but (as it is an L3
node) it decodes the data flow ID based on the "L3-ID" fields
of the received packet.
shows a scenario where MPLS domain nodes ("PE-n" and "P-m") are connected via two Ethernet switches ("ETH-n").
"PE-1" uses the MPLS-specific ID ("MPLS-ID"), but as it
is connected to an Ethernet domain, it has to push an
Ethernet-domain-specific Flow-ID
("ETH-ID") before sending the
packet to "ETH-1". Ethernet switch "ETH-1" can recognize the
data flow based on the "ETH-ID", and it does forwarding toward
"ETH-2". "ETH-2" switches the packet toward the MPLS node
("P-2"). "P-2" must be configured to receive the Ethernet
Flow-ID-specific multicast flow, but (as it is an MPLS node)
it decodes the data flow ID based on the "MPLS-ID" fields of
the received packet.
One can appreciate from the above example that, when the means used for DetNet flow identification
is altered or exported, the means for encoding the sequence number information must similarly
be altered or exported.
Advertising Resources, Capabilities, and Adjacencies
Provisioning of DetNet requires knowledge about:
Details of the DetNet system's capabilities that are required in order to
accurately allocate that DetNet system's resources, as well as other DetNet systems'
resources. This includes, for example, which specific queuing and
shaping algorithms are implemented (),
the number of buffers dedicated for DetNet allocation, and the worst-case
forwarding delay and misordering.
The actual state of a DetNet node's DetNet resources.
The identity of the DetNet system's neighbors and the characteristics of the
link(s) between the DetNet systems, including the latency of
the links (in nanoseconds).
Scaling to Larger Networks
Reservations for individual DetNet flows require considerable state information in
each DetNet node, especially when adequate fault mitigation
() is required. The DetNet Data Plane, in order to
support larger numbers of DetNet flows, must support the aggregation of DetNet flows.
Such aggregated flows can be viewed by the DetNet nodes' Data Plane
largely as individual DetNet flows. Without such aggregation, the per-relay
system may limit the scale of DetNet networks. Example techniques that may be used
include MPLS hierarchy and IP DiffServ Code Points (DSCPs).
Compatibility with Layer 2
Standards providing similar capabilities for bridged networks (only) have
been and are being generated in the IEEE 802 LAN/MAN Standards Committee.
The present architecture describes an abstract model that can be applicable
both at Layer 2 and Layer 3, and over links not defined by IEEE 802. DetNet-enabled end systems and DetNet nodes can be interconnected by
sub-networks, i.e., Layer 2 technologies. These sub-networks will provide
DetNet compatible service for support of DetNet traffic. Examples of
sub-network technologies include MPLS TE, TSN as defined by IEEE 802.1, and a point-to-point OTN
link. Of course, multilayer DetNet systems may be possible too, where one
DetNet appears as a sub-network and provides service to a higher-layer DetNet
system.
Security Considerations
Security considerations for DetNet are described in detail
in . This section considers
exclusively security considerations that are specific to the
DetNet architecture. Security aspects that are
unique to DetNet are those whose aim is to provide the
specific QoS aspects of DetNet, which are
primarily to deliver data flows with extremely low packet
loss rates and bounded end-to-end delivery latency. A DetNet
may be implemented using MPLS and/or IP (including both v4
and v6) technologies and thus inherits the security
properties of those technologies at both the Data Plane and
the Controller Plane. Security considerations for
DetNet are constrained (compared to, for example, the open
Internet) because DetNet is defined to operate only within a
single administrative domain (see ). The primary
considerations are to secure the request and control of
DetNet resources, maintain confidentiality of data
traversing the DetNet, and provide the availability of the
DetNet QoS. To secure the request
and control of DetNet resources, authentication and
authorization can be used for each device connected to a
DetNet domain, most importantly to network controller
devices. Control of a DetNet network may be centralized or
distributed (within a single administrative domain). In the
case of centralized control, precedent for security
considerations as defined for Abstraction and Control of
Traffic Engineered Networks (ACTN) can be found in . In the case of distributed
control protocols, DetNet security is expected to be
provided by the security properties of the protocols in
use. In any case, the result is that manipulation of
administratively configurable parameters is limited to
authorized entities. To maintain confidentiality of
data traversing the DetNet, application flows can be
protected through whatever means is provided by the
underlying technology. For example, encryption may be used,
such as that provided by IPsec , for
IP flows and by MACSec for
Ethernet (Layer 2) flows. DetNet flows are
identified on a per-flow basis, which may provide attackers
with additional information about the data flows (when
compared to networks that do not include per-flow
identification). This is an inherent property of DetNet
that has security implications that should be considered
when determining if DetNet is a suitable technology for any
given use case. To provide uninterrupted
availability of the DetNet QoS, provisions
can be made against DoS attacks and delay attacks. To
protect against DoS attacks, excess traffic due to malicious
or malfunctioning devices can be prevented or mitigated, for
example, through the use of traffic admission control
applied at the input of a DetNet domain as described in
and through the fault-mitigation
methods described in . To
prevent DetNet packets from being delayed by an entity
external to a DetNet domain, DetNet technology definition
can allow for the mitigation of man-in-the-middle attacks,
for example, through use of authentication and authorization
of devices within the DetNet domain. Because DetNet
mechanisms or applications that rely on DetNet can make
heavy use of methods that require precise time
synchronization, the accuracy, availability, and integrity
of time synchronization is of critical importance. Extensive
discussion of this topic can be found in . DetNet use cases are known to
have widely divergent security requirements. The intent of
this section is to provide a baseline for security
considerations that are common to all DetNet designs and
implementations, without burdening individual designs with
specifics of security infrastructure that may not be
germane to the given use case. Designers and implementors of
DetNet systems are expected to take use-case-specific considerations
into account in their DetNet designs
and implementations.
Privacy Considerations
DetNet provides a QoS, and the generic
considerations for such mechanisms apply. In particular, such markings
allow for an attacker to correlate flows or to select particular types
of flow for more detailed inspection.
However, the requirement for every (or almost every) node along the path of
a DetNet flow to identify DetNet flows may present an additional attack
surface for privacy should the DetNet paradigm be found useful in broader
environments.
IANA ConsiderationsThis document has no IANA actions.
Informative ReferencesBufferbloat: Dark Buffers in the InternetCommon Control and Measurement Plane (ccamp)IETFDetNet Data Plane FrameworkThis document provides an overall framework for the Deterministic Networking data plane. It covers concepts and considerations that are generally common to any Deterministic Networking data plane specification.Work in ProgressDetNet Data Plane: IPThis document specifies the Deterministic Networking data plane when operating in an IP packet switched network.Work in ProgressDetNet Data Plane: MPLSThis document specifies the Deterministic Networking data plane when operating over an MPLS Packet Switched Networks.Work in ProgressDeterministic Networking (DetNet) Security ConsiderationsA deterministic network is one that can carry data flows for real- time applications with extremely low data loss rates and bounded latency. Deterministic networks have been successfully deployed in real-time operational technology (OT) applications for some years (for example [ARINC664P7]). However, such networks are typically isolated from external access, and thus the security threat from external attackers is low. IETF Deterministic Networking (DetNet) specifies a set of technologies that enable creation of deterministic networks on IP-based networks of potentially wide area (on the scale of a corporate network) potentially bringing the OT network into contact with Information Technology (IT) traffic and security threats that lie outside of a tightly controlled and bounded area (such as the internals of an aircraft). These DetNet technologies have not previously been deployed together on a wide area IP-based network, and thus can present security considerations that may be new to IP- based wide area network designers. This draft, intended for use by DetNet network designers, provides insight into these security considerations. In addition, this draft collects all security- related statements from the various DetNet drafts (Architecture, Use Cases, etc) into a single location Section 8.Work in ProgressIndustrial communication networks - High availability automation networks - Part 3: Parallel Redundancy Protocol (PRP) and High-availability Seamless Redundancy (HSR)IECIEEE Standard for Local and metropolitan area networks-Media Access Control (MAC) SecurityIEEEIEEE Standard for Local and metropolitan area networks--Audio Video Bridging (AVB) SystemsIEEEIEEE Standard for Local and metropolitan area networks--Frame Replication and Elimination for ReliabilityIEEEIEEE Standard for Local and Metropolitan Area Network--Bridges and Bridged NetworksIEEEIEEE Standard for Local and Metropolitan Area Networks - Virtual Bridged Local Area Networks Amendment 12: Forwarding and Queuing Enhancements for Time-Sensitive StreamsIEEEIEEE Standard for Local and metropolitan area networks -- Bridges and Bridged Networks -- Amendment 26: Frame PreemptionIEEEIEEE Standard for Local and metropolitan area networks -- Bridges and Bridged Networks - Amendment 25: Enhancements for Scheduled TrafficIEEEIEEE Standard for Local and metropolitan area networks--Bridges and Bridged Networks--Amendment 29: Cyclic Queuing and ForwardingIEEETime-Sensitive Networking (TSN) Task GroupIEEEIEEE Standard for EthernetIEEEIEEE Standard for Ethernet Amendment 5: Specification and Management Parameters for Interspersing Express TrafficIEEEResource ReSerVation Protocol (RSVP) -- Version 1 Functional SpecificationThis memo describes version 1 of RSVP, a resource reservation setup protocol designed for an integrated services Internet. RSVP provides receiver-initiated setup of resource reservations for multicast or unicast data flows, with good scaling and robustness properties. [STANDARDS-TRACK]An Architecture for Differentiated ServicesThis document defines an architecture for implementing scalable service differentiation in the Internet. This memo provides information for the Internet community.Congestion Control PrinciplesThe goal of this document is to explain the need for congestion control in the Internet, and to discuss what constitutes correct congestion control. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.The Addition of Explicit Congestion Notification (ECN) to IPThis memo specifies the incorporation of ECN (Explicit Congestion Notification) to TCP and IP, including ECN's use of two bits in the IP header. [STANDARDS-TRACK]RSVP-TE: Extensions to RSVP for LSP TunnelsThis document describes the use of RSVP (Resource Reservation Protocol), including all the necessary extensions, to establish label-switched paths (LSPs) in MPLS (Multi-Protocol Label Switching). Since the flow along an LSP is completely identified by the label applied at the ingress node of the path, these paths may be treated as tunnels. A key application of LSP tunnels is traffic engineering with MPLS as specified in RFC 2702. [STANDARDS-TRACK]RTP: A Transport Protocol for Real-Time ApplicationsThis memorandum describes RTP, the real-time transport protocol. RTP provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services. RTP does not address resource reservation and does not guarantee quality-of- service for real-time services. The data transport is augmented by a control protocol (RTCP) to allow monitoring of the data delivery in a manner scalable to large multicast networks, and to provide minimal control and identification functionality. RTP and RTCP are designed to be independent of the underlying transport and network layers. The protocol supports the use of RTP-level translators and mixers. Most of the text in this memorandum is identical to RFC 1889 which it obsoletes. There are no changes in the packet formats on the wire, only changes to the rules and algorithms governing how the protocol is used. The biggest change is an enhancement to the scalable timer algorithm for calculating when to send RTCP packets in order to minimize transmission in excess of the intended rate when many participants join a session simultaneously. [STANDARDS-TRACK]Security Architecture for the Internet ProtocolThis document describes an updated version of the "Security Architecture for IP", which is designed to provide security services for traffic at the IP layer. This document obsoletes RFC 2401 (November 1998). [STANDARDS-TRACK]A Path Computation Element (PCE)-Based ArchitectureConstraint-based path computation is a fundamental building block for traffic engineering systems such as Multiprotocol Label Switching (MPLS) and Generalized Multiprotocol Label Switching (GMPLS) networks. Path computation in large, multi-domain, multi-region, or multi-layer networks is complex and may require special computational components and cooperation between the different network domains.This document specifies the architecture for a Path Computation Element (PCE)-based model to address this problem space. This document does not attempt to provide a detailed description of all the architectural components, but rather it describes a set of building blocks for the PCE architecture from which solutions may be constructed. This memo provides information for the Internet community.MPLS Transport Profile (MPLS-TP) Survivability FrameworkNetwork survivability is the ability of a network to recover traffic delivery following failure or degradation of network resources. Survivability is critical for the delivery of guaranteed network services, such as those subject to strict Service Level Agreements (SLAs) that place maximum bounds on the length of time that services may be degraded or unavailable.The Transport Profile of Multiprotocol Label Switching (MPLS-TP) is a packet-based transport technology based on the MPLS data plane that reuses many aspects of the MPLS management and control planes.This document comprises a framework for the provision of survivability in an MPLS-TP network; it describes recovery elements, types, methods, and topological considerations. To enable data-plane recovery, survivability may be supported by the control plane, management plane, and by Operations, Administration, and Maintenance (OAM) functions. This document describes mechanisms for recovering MPLS-TP Label Switched Paths (LSPs). A detailed description of pseudowire recovery in MPLS-TP networks is beyond the scope of this document.This document is a product of a joint Internet Engineering Task Force (IETF) / International Telecommunication Union Telecommunication Standardization Sector (ITU-T) effort to include an MPLS Transport Profile within the IETF MPLS and Pseudowire Emulation Edge-to-Edge (PWE3) architectures to support the capabilities and functionalities of a packet-based transport network as defined by the ITU-T. This document is not an Internet Standards Track specification; it is published for informational purposes.Packet Pseudowire Encapsulation over an MPLS PSNThis document describes a pseudowire mechanism that is used to transport a packet service over an MPLS PSN in the case where the client Label Switching Router (LSR) and the server Provider Edge equipments are co-resident in the same equipment. This pseudowire mechanism may be used to carry all of the required layer 2 and layer 3 protocols between the pair of client LSRs. [STANDARDS-TRACK]Software-Defined Networking: A Perspective from within a Service Provider EnvironmentSoftware-Defined Networking (SDN) has been one of the major buzz words of the networking industry for the past couple of years. And yet, no clear definition of what SDN actually covers has been broadly admitted so far. This document aims to clarify the SDN landscape by providing a perspective on requirements, issues, and other considerations about SDN, as seen from within a service provider environment.It is not meant to endlessly discuss what SDN truly means but rather to suggest a functional taxonomy of the techniques that can be used under an SDN umbrella and to elaborate on the various pending issues the combined activation of such techniques inevitably raises. As such, a definition of SDN is only mentioned for the sake of clarification.Security Requirements of Time Protocols in Packet Switched NetworksAs time and frequency distribution protocols are becoming increasingly common and widely deployed, concern about their exposure to various security threats is increasing. This document defines a set of security requirements for time protocols, focusing on the Precision Time Protocol (PTP) and the Network Time Protocol (NTP). This document also discusses the security impacts of time protocol practices, the performance implications of external security practices on time protocols, and the dependencies between other security services and time synchronization.Software-Defined Networking (SDN): Layers and Architecture TerminologySoftware-Defined Networking (SDN) refers to a new approach for network programmability, that is, the capacity to initialize, control, change, and manage network behavior dynamically via open interfaces. SDN emphasizes the role of software in running networks through the introduction of an abstraction for the data forwarding plane and, by doing so, separates it from the control plane. This separation allows faster innovation cycles at both planes as experience has already shown. However, there is increasing confusion as to what exactly SDN is, what the layer structure is in an SDN architecture, and how layers interface with each other. This document, a product of the IRTF Software-Defined Networking Research Group (SDNRG), addresses these questions and provides a concise reference for the SDN research community based on relevant peer-reviewed literature, the RFC series, and relevant documents by other standards organizations.Using IEEE 802.15.4e Time-Slotted Channel Hopping (TSCH) in the Internet of Things (IoT): Problem StatementThis document describes the environment, problem statement, and goals for using the Time-Slotted Channel Hopping (TSCH) Medium Access Control (MAC) protocol of IEEE 802.14.4e in the context of Low-Power and Lossy Networks (LLNs). The set of goals enumerated in this document form an initial set only.IETF Recommendations Regarding Active Queue ManagementThis memo presents recommendations to the Internet community concerning measures to improve and preserve Internet performance. It presents a strong recommendation for testing, standardization, and widespread deployment of active queue management (AQM) in network devices to improve the performance of today's Internet. It also urges a concerted effort of research, measurement, and ultimate deployment of AQM mechanisms to protect the Internet from flows that are not sufficiently responsive to congestion notification.Based on 15 years of experience and new research, this document replaces the recommendations of RFC 2309.IS-IS Path Control and ReservationIEEE 802.1Qca Path Control and Reservation (PCR) specifies explicit path control via IS-IS in Layer 2 networks in order to move beyond the shortest path capabilities provided by IEEE 802.1aq Shortest Path Bridging (SPB). IS-IS PCR provides capabilities for the establishment and control of explicit forwarding trees in a Layer 2 network domain. This document specifies the sub-TLVs for IS-IS PCR.Proportional Integral Controller Enhanced (PIE): A Lightweight Control Scheme to Address the Bufferbloat ProblemBufferbloat is a phenomenon in which excess buffers in the network cause high latency and latency variation. As more and more interactive applications (e.g., voice over IP, real-time video streaming, and financial transactions) run in the Internet, high latency and latency variation degrade application performance. There is a pressing need to design intelligent queue management schemes that can control latency and latency variation, and hence provide desirable quality of service to users.This document presents a lightweight active queue management design called "PIE" (Proportional Integral controller Enhanced) that can effectively control the average queuing latency to a target value. Simulation results, theoretical analysis, and Linux testbed results have shown that PIE can ensure low latency and achieve high link utilization under various congestion situations. The design does not require per-packet timestamps, so it incurs very little overhead and is simple enough to implement in both hardware and software.MPLS-TP Shared-Ring Protection (MSRP) Mechanism for Ring TopologyThis document describes requirements, architecture, and solutions for MPLS-TP Shared-Ring Protection (MSRP) in a ring topology for point- to-point (P2P) services. The MSRP mechanism is described to meet the ring protection requirements as described in RFC 5654. This document defines the Ring Protection Switching (RPS) protocol that is used to coordinate the protection behavior of the nodes on an MPLS ring.Controlled Delay Active Queue ManagementThis document describes CoDel (Controlled Delay) -- a general framework that controls bufferbloat-generated excess delay in modern networking environments. CoDel consists of an estimator, a setpoint, and a control loop. It requires no configuration in normal Internet deployments.Segment Routing ArchitectureSegment Routing (SR) leverages the source routing paradigm. A node steers a packet through an ordered list of instructions, called "segments". A segment can represent any instruction, topological or service based. A segment can have a semantic local to an SR node or global within an SR domain. SR provides a mechanism that allows a flow to be restricted to a specific topological path, while maintaining per-flow state only at the ingress node(s) to the SR domain.SR can be directly applied to the MPLS architecture with no change to the forwarding plane. A segment is encoded as an MPLS label. An ordered list of segments is encoded as a stack of labels. The segment to process is on the top of the stack. Upon completion of a segment, the related label is popped from the stack.SR can be applied to the IPv6 architecture, with a new type of routing header. A segment is encoded as an IPv6 address. An ordered list of segments is encoded as an ordered list of IPv6 addresses in the routing header. The active segment is indicated by the Destination Address (DA) of the packet. The next active segment is indicated by a pointer in the new routing header.Framework for Abstraction and Control of TE Networks (ACTN)Traffic Engineered (TE) networks have a variety of mechanisms to facilitate the separation of the data plane and control plane. They also have a range of management and provisioning protocols to configure and activate network resources. These mechanisms represent key technologies for enabling flexible and dynamic networking. The term "Traffic Engineered network" refers to a network that uses any connection-oriented technology under the control of a distributed or centralized control plane to support dynamic provisioning of end-to- end connectivity.Abstraction of network resources is a technique that can be applied to a single network domain or across multiple domains to create a single virtualized network that is under the control of a network operator or the customer of the operator that actually owns the network resources.This document provides a framework for Abstraction and Control of TE Networks (ACTN) to support virtual network services and connectivity services.Deterministic Networking Problem StatementThis paper documents the needs in various industries to establish multi-hop paths for characterized flows with deterministic properties.Deterministic Networking Use CasesThis document presents use cases for diverse industries that have in common a need for "deterministic flows". "Deterministic" in this context means that such flows provide guaranteed bandwidth, bounded latency, and other properties germane to the transport of time-sensitive data. These use cases differ notably in their network topologies and specific desired behavior, providing as a group broad industry context for Deterministic Networking (DetNet). For each use case, this document will identify the use case, identify representative solutions used today, and describe potential improvements that DetNet can enable.Traffic Engineering Architecture and Signaling (teas)IETFAn Architecture for IPv6 over the TSCH mode of IEEE 802.15.4This document describes a network architecture that provides low- latency, low-jitter and high-reliability packet delivery. It combines a high-speed powered backbone and subnetworks using IEEE 802.15.4 time-slotted channel hopping (TSCH) to meet the requirements of LowPower wireless deterministic applications.Work in ProgressAcknowledgementsThe authors wish to thank Lou Berger, David Black, Stewart
Bryant, Rodney Cummings, Ethan Grossman, Craig Gunther, Marcel
Kiessling, Rudy Klecka, Jouni Korhonen, Erik Nordmark, Shitanshu
Shah, Wilfried Steiner, George Swallow, Michael Johas Teener, Pat
Thaler, Thomas Watteyne, Patrick Wetterwald, Karl Weber, and Anca
Zamfir for their various contributions to this work.
Authors' AddressesHuawei3101 Rio WaySpring ValleyCalifornia91977United States of America+1 925 980 6430nfinn@nfinnconsulting.comCisco SystemsVillage d'Entreprises Green Side400, Avenue de RoumanilleBatiment T3Biot - Sophia Antipolis06410France+33 4 97 23 26 34pthubert@cisco.comEricssonMagyar tudosok korutja 11BudapestHungary1117balazs.a.varga@ericsson.comEricssonMagyar tudosok korutja 11BudapestHungary1117janos.farkas@ericsson.com