IS-IS Support for Openfabric

Spine and leaf fabrics are often used in large scale data centers; in this application, they are commonly called a fabric because of their regular structure and predictable forwarding and convergence properties. This document describes modifications to the IS-IS protocol to enable it to run efficiently on a large scale spine and leaf fabric, openfabric. The goals of this control plane are: Provide a full view of the topology from a single point in the network to simplify operations Minimize configuration of each IS in the network Optimize the operation of IS-IS within a spine and leaf fabric to enable scaling

The following people have contributed to this draft: Nikos Triantafillis (reflected flooding optimization), Ivan Pepelnjak (three stage fabric modifications), Hannes Gredler (do not reflood optimizations), Les Ginsberg (capabilities encoding, circuit local reflooding), Naiming Shen (capabilities encoding, circuit local reflooding), Uma Chunduri (failure mode suggestions, flooding), Nick Russo, and Rodny Molina. See , , and for similar solutions in the Mobile Ad Hoc Networking (MANET) solution space.

In building any scalable system, it is often best to begin by removing what is not needed. In this spirit, openfabric implementations MAY remove the following from IS-IS: External metrics. There is no need for external metrics in large scale spine and leaf fabrics; it is assumed that metrics will be properly configured by the operator to account for the correct order of route preference at any route redistribution point. Tags and traffic engineering processing. Openfabric is only designed to provide topology and reachability information. It is not designed to provide for traffic engineering, route preference through tags, or other policy mechanisms. It is assumed that all routing policy will be provided through an overlay system which communicates directly with each IS in the fabric, such as PCEP or I2RS. Traffic engineering is assumed to be provided through Segment Routing (SR).

To create a scalable link state fabric, openfabric includes the following: A slightly modified adjacency formation process. Mechanisms for determining which tier within a spine and leaf fabric in which the IS is located. A mechanism that reduces flooding to the minimum possible, while still ensuring complete database synchronization among the intermediate systems within the fabric. Three general requirements are placed here; more specific requirements are considered in the following sections. Openfabric implementations: MUST support and enable hostname advertisement by default if a hostname is configured on the intermediate system. SHOULD support , purge originator identification for IS-IS. MUST NOT be mixed with standard IS-IS implementations in operational deployments. Openfabric and standard IS-IS implementations SHOULD be treated as two separate protocols.

The following spine and leaf fabric will be used to describe these modifications.

To reduce confusion (spine and leaf fabrics are difficult to draw in plain text art), this diagram does not contain the connections between devices. The reader should assume that each device in a given layer is connected to every device in the layer above it. For instance: 5A is connected to 4A, 4B, 4C, 4D, 4E, and 4F 5B is connected to 4A, 4B, 4C, 4D, 4E, and 4F 4A is connected to 3A, 3B, 3C, 3D, 3E, 3F, 5A, 5B, 5C, 5D, 5E, and 5F 4B is connected to 3A, 3B, 3C, 3D, 3E, 3F, 5A, 5B, 5C, 5D, 5E, and 5F etc. The tiers or stages of the fabric are also marked for easier reference. T0 is assumed to be connected to application servers, or rather they are Top of Rack (ToR) intermediate systems. The remaining tiers, T1 and T2, are connected only to the fabric itself. Note there are no "cross links," or "east west" links in the illustrated fabric. The fabric locality detection mechanism described here will not work if there are cross links running east/west through the fabric. Locality detection may be possible in such a fabric; this is an area for further study.

Because Openfabric operates in a tightly controlled data center environment, various modifications can be made to the IS-IS neighbor formation process to increase efficencicy and simplify the protocol. Specifically, Openfabric implementations SHOULD support , section 4, hello padding for IS-IS. Variable hello padding SHOULD NOT be used, as data center fabrics are built using high speed links on which padded hellos will have little performance impact. Further modifications to the neighbor formation process are considered in the following sections.

Openfabric is designed to work in a single flooding domain over a single data center fabric at the scale of thousands of routers with hundreds of thousands of routes (so a moderate scale in router and route count terms). Because of the way Openfabric optimizes operation in this environment, it is not necessary nor desirable to build multiple flooding domains. For instance, the flooding optimizations described later this document require a full view of the topology, as does any proposed overlay to inject policy into the forwarding plane. In light of this, the following changes SHOULD BE to IS-IS implemetations to support Openfabric: IIH PDU 16 (level 2 broadcast circuit hello) should be the only IIH PDU type transmitted (see section 9.6 of [ISO10589] and section 4.1 of ) In IIH PDU 16 (level 2 broadcast circuit hello), the Circuit Type field should be set to 2 (see section 9.6 of [ISO10589]) Support for IIH PDU 15 (level 1 broadcast hello) should be removed (see section 9.5 of [ISO10589]) Support for IIH PDU 17 (point-to-pint hello) should be removed (see section 9.7 of [ISO10589])

Data center network fabrics only contain point-to-point links; because of this, there is no reason to support any broadcast link types, nor to support the Designated Intermediate System processing, including pseudonode creation. In light ot his, processing related to sections 7.2.3 (broadcast networks), 7.3.8 (generation of level 1 pseudonode LSPs), 7.3.10 (generation of level 2 pseudonode LSPs), and section 8.4.5 (LAN designated intermediate systems) in [ISO10589] SHOULD BE removed.

It is important that two way connectivity be established before synchronizing the link state database, or routing through a link in a data center fabric. To reject optical failures that cause a one way connection between two routers, fabricDC must support the three way handshake mechanism described in .

While adjacency formation is not considered particularly burdensome in IS-IS, it is still useful to reduce the amount of state transferred across the network when connecting a new IS to the fabric. Any such optimization is bound to present a tradeoff between several factors; the mechanism described here increases the amount of time required to form adjacencies slightly in order to reduce the total state carried across the network. The process is: An IS connected to the fabric will send hellos on all links. The IS will only complete the three-way handshake with one newly discovered neighbor; this would normally be the first neighbor which sends the newly connected intermediate system's ID back in the three-way handshake process. The IS will complete its database exchange with this one newly adjacent neighbor. Once this process is completed, the IS will continue processing the remaining neighbors as normal. This process allows each IS newly added to the fabric to exchange a full table once; a very minimal amount of information will be transferred with the remaining neighbors to reach full synchronization.

IS-IS describes the topology in two different sets of TLVs; the first describes the set of neighbors connected to an IS, the second describes the set of reachable destination connected to an IS. There are two different forms of both of these descriptions, one of which carries what are widely called narrow metrics, the other of which carries what are widely called wide metrics. In a tightly controlled data center fabric implementation, such as the ones Openfabric is designed to support, no IS that supports narrow metrics will ever be deployed or supported; hence there is no reason to support any metric type other than wide metrics. The Level 2 Link State PDU (type 20 in section 9.9 of [ISO10589]) and the scoped flooding PDU (type 10 in section 3.1 of ) SHOULD BE the only PDU types used to carry link state information in a Openfabric implementation Processing related to the Level 1 Link State PDU (type 18) MAY BE removed from Openfabric implementations (see section 9.8 of [ISO10589]) Neighbor reachability MUST BE carried in TLV type 22 (see section 3 of ) IPv4 reachability SHOULD BE carried in TLV type 135 (see section 4 of ), or TLV type 235 for multitopology implementations (see ) IPv6 reachability SHOULD BE carried in TLV type 236 (see ), or TLV type 237 for multitopology implemenations (see ) Processing related to the neighbor reachability TLV (type 2, see sections 9.8 and 9.9 of [ISO10589]) SHOULD BE removed Processing related to the narrow metric IP reachability TLV (types 128 and 130) SHOULD BE removed In order to support segment routing, Openfabric needs to be able to support the advertisement of a Prefix-SID tied to a local loopback address assigned to the IS. The configuration of the label to advertise MAY BE manually configured for the moment or determined through autoconfiguration. A Prefix-SID SHOULD BE advertised if a local label is configured using the Prefix Segment Identifier sub-TLV (see section 2.1 of ).

The tier to which a IS is connected is useful to enable autoconfiguration of intermediate systems connected to the fabric and to reduce flooding. Once the tier of an intermediate system within the fabric has been determined, it MUST be advertised using the 4 bit Tier field described in section 3.3 of . This section describes two mechanisms for determining the tier at which a IS is connected in the fabric in several steps.

The first method begins with one of the T0 intermediate systems advertising its location in the fabric. This information can either be obtained through: A single T0 intermediate system is manually configured to advertise 0x00 in their IS reachability tier sub-TLV, indicating they are at the edge of the fabric (a ToR IS). The T0 intermediate systems detect they are T0 through the presence connected hosts (i.e. through a request for address assignment or some other means). If such detection is used, and the IS determines it is located at T0, it should advertise 0x00 in its IS reachability tier sub-TLV. The second method above SHOULD be used with care, as it may not be secure, and it may not work in all data center environments. For instance, if a host is mistakenly (or intentionally, as a form of attack) attached to a spine IS, or a request for address assignment is transmitted to a spine IS during the bootup phase of the device or fabric, it is possible to cause a spine IS to advertise itself as a T0. Unless the autodetection of the T0 devices is secured, the manual mechanism SHOULD BE used (configuring at least one T0 device manually). Given at least one T0 device is advertising its tier number, the remaining intermediate systems calculate their tier number as follows: The local IS calculates an SPT (using SPF) setting the cost of every link to 1; this effectively calculates a topology only view of the network, without considering any configured link costs Find the closest IS advertising a tier number of 0 in the Spine Leaf extension sub-TLV; call this node A, and set FD to this cost Calculate an SPT (using SPF) from the perspective of A (above), and setting the cost of every link to 1; the maximum cost to any node should be 2 for a 3 stage fabric, 4 for a 5 stage fabric, etc. Choose any node that is a maximum metric from A (above); call this IS B Find the cost to B on the locally calculated SPT from the first step; call this TD Calculate the tier number of the local node by subtracting FD from TD In the example network, assume 5A is manually configured as a T0, and is advertising its tier number. From here: From 1A the path to 5A is 4 hops; this is FD Run SPF from the perspective of 5A with all link metrics set to 1 The maximum path length is 4; 1F is one such node; set this node to B, and set TD to 4 TD - FD is 0 at 1A, so 1A is T0, or a ToR This process will work for any spine and leaf fabric without "cross links."

In some fabrics, it is possible to calculate which intermediate systems are at T0 using a modified Shortest Path First (SPF) calculation. Specifically, if the fabric is configured in five stages, as shown in the example network, and is not some form of butterfly, Benes, or a three stage fabric, it is possible to calcualte if an IS is at T0 using the following process: Calculate a Shortest Path Tree (SPT) for the entire network with all link metrics set to 1; this has the effect of calculating a tree based only on hop count Find one node that is the farthest from the local node in the resulting tree; call this node F, and the distance to this node FD Calculate an SPT for the entire network with all link metrics set to 1 from the perspective of F; call this TD If FD == TD, and TD >= 4, this is a greater than three stage fabric; the local device SHOULD advertise 0x00 in its IS reachability tier sub-TLV. For instance, in the diagram above, 1A would: Calculate an SPT with all link metrics set to 1; on this SPT, 5A through 5F would all have a distance of 4 Select one of these nodes as F; assume 5F is chosen as F Set FD to 4, the distance to 5F Run SPF from the perspective of 5F with all link metrics set to 1 Set TD to 4, the cost from 5F to 1A TD - FD == 0, so 1A is at T0, and is a ToR For the remaining intermediate systems to determine which tier they are situated on, they perform the following calculation: Calculate a Shortest Path Tree (SPT) for the entire network with all link metrics set to 1; this has the effect of calculating a tree based only on hop count Find one node that is the farthest from the local node in the resulting tree; call this node F, and the distance to this node FD Calculate an SPT for the entire network with all link metrics set to 1 from the perspective of F; call this TD The IS SHOULD advertise (TD - FD) in its IS reachability tier sub-TLV. For example, in the above five stage fabric, 3B would: Calculate an SPT with all link metrics set to 1; on this SPT, 5A through 5F and 1A through 1F would all have a cost of 2 Select one of these nodes as F; assume 5F is chosen as F Set FD to 2, the distance to 5F Run SPF from the perspective of 5F with all link metrics set to 1 Set TD to 4, the cost from 5F to 1A TD - FD == 2, so 1A is at T2, and is a spine switch

Flooding is perhaps the most challenging scaling issue for a link state protocol running on a dense, large scale fabric. To reduce the flooding of link state information in the form of Link State Protocol Data Units (LSPs), Openfabric takes advantage of information already available in the link state protocol, the list of the local intermediate system's neighbor's neighbors, and the fabric locality computed above. The following tables are required to compute a set of reflooders: Neighbor List (NL) list: The set of neighbors Neighbor's Neighbors (NN) list: The set of neighbor's neighbors; this can be calculated by running SPF truncated to two hops Do Not Reflood (DNR) list: The set of neighbors who should have LSPs (or fragments) who should not reflood LSPs Reflood (RF) list: The set of neighbors who should flood LSPs (or fragments) to their adjacent neighbors to ensure synchronization NL is set to contain all neighbors, and sorted deterministically (for instance, from the highest IS identifier to the lowest). All intermediate systems within a single fabric SHOULD use the same mechanism for sorting the NL list. NN is set to contain all neighbor's neighbors, or all intermediate systems that are two hops away, as determined by performing a truncated SPF. The DNR and RF tables are initially empty. To begin, the following steps are taken to reduce the size of NN and NL: Move any IS in NL with its tier (or fabric location) set to T0 to DNR Remove all intermediate systems from NL and NN that in the shortest path to the IS that originated the LSP Then, for every IS in NL: If the current entry in NL is connected to any entries in NN: Move the IS to RF Remove the intermediate systems connected to the IS from NN Else move the IS to DNR When flooding, LSPs transmitted to adjacent neighbors on the RF list will be transmitted normally. Adjacent intermediate systems on this list will reflood received LSPs into the next stage of the topology, ensuring database synchronization. LSPs transmitted to adjacent neighbors on the DNR list, however, MUST be transmitted using a circuit scope PDU as described in .

It is possible in some failure modes for flooding to be incomplete because of the flooding optimizations outlined. Specifically, if a reflooder fails, or is somehow disconnected from all the links across which it should be reflooding, it is possible an LSP is only partially flooded through the fabric. To prevent such situations, any IS receiving an LSP transmitted using DNR SHOULD: Set a short timer; the default should be less than one second When the timer expires, send a Complete Sequence Number Packet (CSNP) to all neighbors Process any Partial Sequence Number Packets (PSNPs) as required to resynchronize If a resynchronization is required, notify the network operator through a network management system

In order to reduce the amount of control plane state carried on large scale spine and leaf fabrics, openfabric implementations SHOULD NOT advertise reachability for transit links. These links MAY remain unnumbered, as IS-IS does not require layer 3 IP addresses to operate. Each IS SHOULD be configured with a single loopback address, which is assigned an IPv6 address, to provide reachability to intermediate systems which make up the fabric.

In data center fabrics, ToR intermediate systems SHOULD NOT be used to transit between two T1 (or above) spine intermediate systems. The simplest way to prevent this is to set the overload bit for all the LSPs originated from T0 intermediate systems. However, this solution would have the unfortunate side effect of causing all reachability beyond any T0 IS to have the same metric, and many implementations treat a set overload bit as a metric of 0xFFFF in calculating the Shortest Path Tree (SPT). This document proposes an alternate solution which preserves the leaf node metric, while still avoiding transiting T0 intermediate systems. Specifically, all T0 intermediate systems SHOULD advertise their metric to reach any T1 adjacent neighbor with a cost of 0XFFE. T1 intermediate systems, on the other hand, will advertise T0 intermediate systems with the actual interface cost used to reach the T0 IS. Hence, links connecting T0 and T1 intermediate systems will be advertised with an asymmetric cost that discourages transiting T0 intermediate systems, while leaving reachability to the destinations attached to T0 devices the same.

While schemes may be designed so reachability information can be aggregated in Openfabric deployments, this is not a recommended configuraiton.

This document outlines modifications to the IS-IS protocol for operation on large scale data center fabrics. While it does add new TLVs, and some local processing changes, it does not add any new security vulnerabilities to the operation of IS-IS. However, openfabric implementations SHOULD implement IS-IS cryptographic authentication, as described in , and should enable other security measures in accordance with best common practices for the IS-IS protocol. If T0 intermediate systems are auto-detected using information outside Openfabric, it is possible to attack the calucations used for flooding reduction and auto-configuration of intermediate systems. For instance, if a request for an address pool is used as an indicator of an attached host, and hence receiving such a request causes an intermediate system to advertise itself as T0, it is possible for an attacker (or a simple mistake) to cause auto-configuration to fail. Any such auto-detection mechanims SHOULD BE secured using appropriate techniques, as described by any protocols or mechanisms used.