Individual Submission T. Player Internet-Draft Spirent Communications Intended status: Informational D. Newman Expires: February 15, 2010 Network Test August 14, 2009 Benchmarking Methodology for Data Center Bridging Devices draft-player-dcb-benchmarking-00.txt Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on February 15, 2010. Copyright Notice Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Player & Newman Expires February 15, 2010 [Page 1] Internet-Draft DCB Benchmarking August 2009 Abstract Existing benchmarking methodologies are based on the assumption that networking devices will drop network traffic at their performance limits. Data Center Bridging (DCB) devices, however, will attempt to throttle network endpoints before those limits are reached in order to minimize the probability of frame loss in the network. Hence, existing methodologies based around frame loss are inappropriate for DCB devices. This document takes the basic benchmarking ideas based on loss and extends them to support "lossless" Ethernet devices. Player & Newman Expires February 15, 2010 [Page 2] Internet-Draft DCB Benchmarking August 2009 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 5 3. General Considerations . . . . . . . . . . . . . . . . . . . . 6 3.1. Test Traffic . . . . . . . . . . . . . . . . . . . . . . . 6 4. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 7 5. Test Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.1. Test Traffic . . . . . . . . . . . . . . . . . . . . . . . 9 5.1.1. Traffic Classification . . . . . . . . . . . . . . . . 9 5.1.2. Traffic Generation . . . . . . . . . . . . . . . . . . 9 5.1.3. Trial Duration . . . . . . . . . . . . . . . . . . . . 9 5.1.4. Frame Formats . . . . . . . . . . . . . . . . . . . . 9 5.1.5. Frame Measurements . . . . . . . . . . . . . . . . . . 10 5.1.6. Frame Sizes . . . . . . . . . . . . . . . . . . . . . 10 6. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . . 12 6.1. Queueput . . . . . . . . . . . . . . . . . . . . . . . . . 12 6.1.1. Objective . . . . . . . . . . . . . . . . . . . . . . 12 6.1.2. Setup Parameters . . . . . . . . . . . . . . . . . . . 12 6.1.3. Procedure . . . . . . . . . . . . . . . . . . . . . . 12 6.1.4. Measurements . . . . . . . . . . . . . . . . . . . . . 13 6.1.5. Reporting Format . . . . . . . . . . . . . . . . . . . 13 6.2. Maximum Forwarding Rate . . . . . . . . . . . . . . . . . 13 6.2.1. Objective . . . . . . . . . . . . . . . . . . . . . . 13 6.2.2. Setup Parameters . . . . . . . . . . . . . . . . . . . 13 6.2.3. Procedure . . . . . . . . . . . . . . . . . . . . . . 14 6.2.4. Measurements . . . . . . . . . . . . . . . . . . . . . 14 6.2.5. Reporting Format . . . . . . . . . . . . . . . . . . . 14 6.3. Back-off . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.3.1. Objective . . . . . . . . . . . . . . . . . . . . . . 14 6.3.2. Setup Parameters . . . . . . . . . . . . . . . . . . . 14 6.3.3. Procedure . . . . . . . . . . . . . . . . . . . . . . 15 6.3.4. Measurements . . . . . . . . . . . . . . . . . . . . . 15 6.3.5. Reporting Format . . . . . . . . . . . . . . . . . . . 15 6.4. Back-to-Back . . . . . . . . . . . . . . . . . . . . . . . 16 6.4.1. Objective . . . . . . . . . . . . . . . . . . . . . . 16 6.4.2. Setup Parameters . . . . . . . . . . . . . . . . . . . 16 6.4.3. Procedure . . . . . . . . . . . . . . . . . . . . . . 16 6.4.4. Measurements . . . . . . . . . . . . . . . . . . . . . 16 6.4.5. Reporting Format . . . . . . . . . . . . . . . . . . . 17 7. Security Considerations . . . . . . . . . . . . . . . . . . . 18 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 9. Normative References . . . . . . . . . . . . . . . . . . . . . 20 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 21 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 22 Player & Newman Expires February 15, 2010 [Page 3] Internet-Draft DCB Benchmarking August 2009 1. Introduction This document is intended to provide a methodology for benchmarking Data Center Ethernet (DCB) switching devices that support Priority- based Flow Control (PFC) as defined in [802.1Qbb]. It extends the methodologies already defined in [RFC2544] and [RFC2889]. This RFC primarily deals with devices that use priority-based flow control (PFC), as defined in [802.1Qbb], to actively manage the transmission rate of network endpoints in order to minimize forwarding delay and frame loss. Player & Newman Expires February 15, 2010 [Page 4] Internet-Draft DCB Benchmarking August 2009 2. Requirements The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Player & Newman Expires February 15, 2010 [Page 5] Internet-Draft DCB Benchmarking August 2009 3. General Considerations 3.1. Test Traffic The lock-step traffic pattern, as described in section 5.1.3 of [RFC2889], is specifically NOT required for DCB testing for two reasons: 1. Such patterns are unrealistic for high speed Ethernet devices due to the transmission clock variance allowed by the IEEE 802.3 Ethernet specification. 2. Flow control mechanisms would quickly break such patterns when activated. Player & Newman Expires February 15, 2010 [Page 6] Internet-Draft DCB Benchmarking August 2009 4. Terminology As the terminology used by [RFC4689] is specific to IP layer testing, a number of existing terms require clarification when used in the DCB benchmarking context. Additionally, a number of new terms are also presented to clarify concepts not clearly defined within the scope of [RFC4689]. Classification: As stated in [RFC4689], the selection of packets according to defined rules. In the context of DCB benchmarking, the Classification criterion is the value of the 802.1p priority code point field in the 802.1Q VLAN header of an Ethernet frame. Classification Group: A collection of traffic streams that belong to a single Classification. A Conformance Vector MAY be associated with a Classification Group. Classification Profile: The set of all Classification Groups involved in a benchmarking test. Conformance Vector: A set of measurable stream result bounds, e.g. latency, jitter, sequencing, etc., that specify whether a frame is Conformant or Nonconformant. Congestion Management: In the context of DCB benchmarking, Congestion Management occurs when the DUT/SUT transmits priority- based flow control (PFC) Pause frames. Forwarding Congestion: In the context of DCB benchmarking, Forwarding Congestion is extended to include PFC pause frame transmissions from the DUT. Intended Load: In this document, the Intended Load refers to the summation of the Intended Vectors for all Classification Groups. Offered Load: In this document, the Offered Load refers to the summation of the Offered Vectors for all Classification Groups. Queue Congestion: Queue congestion occurs when a DUT/SUT uses Congestion Management on a discrete set of code points. The congestion managed code points correspond to the congested queue in the DUT/SUT. Queueput: The maximum Offered Load than can be transmitted into a DUT/SUT such that every transmitted frame matches a specific Classification rule, the DUT/SUT does NOT use priority-based flow control mechanisms to manage the ingress traffic rate, and all ingress frames are forwarded to the correct egress port. A DUT Player & Newman Expires February 15, 2010 [Page 7] Internet-Draft DCB Benchmarking August 2009 may have a different Queueput rate for each configured Classification. Player & Newman Expires February 15, 2010 [Page 8] Internet-Draft DCB Benchmarking August 2009 5. Test Setup This document extends the general test setup described in section 3 of [RFC2889] and section 6 of [RFC2544] to the benchmarking of Data Center Ethernet switching devices. [RFC2889] and [RFC2544] describe benchmarking methodologies for networking devices that intentionally drop frames at their performance limits. In DCB networks, the DUT will transmit PFC Pause frames as a Congestion Management method to throttle network endpoints, thus minimizing the probability of frame loss in the network. 5.1. Test Traffic 5.1.1. Traffic Classification Since DCB devices are expected to support multiple traffic Classifications, it is RECOMMENDED to benchmark DCB devices with multiple Classification Groups. 5.1.2. Traffic Generation When generating multiple traffic classifications from the same test port, traffic from each class MUST be interleaved in a round-robin fashion. 5.1.3. Trial Duration The recommended trial duration is 300 seconds, however other durations MAY be used. Additionally, a running trial MAY be aborted once the test tool can tell that the currently running trial has failed. Two examples would be if a frame does not conform to the Conformance Vector of that frame's Classification Group, or if the test tool detects loss on a lossless queue. 5.1.4. Frame Formats This testing document does not mandate the use of any particular frame format for testing. Any frame that can be legally forwarded by the DUT/SUT MAY be used provided that the test instrument can make the following distinctions for each frame: 1. The test tool MUST be able to distinguish test frames from non- test frames. 2. The test tool MUST be able to determine whether each test frame is forwarded to the correct egress port. Player & Newman Expires February 15, 2010 [Page 9] Internet-Draft DCB Benchmarking August 2009 3. The test tool MUST be able to determine whether each received frame conforms or does not conform to the Conformance Vector of the frame's Classification Group. 5.1.5. Frame Measurements Packet Conformance MUST be determined for each and every test frame. The method specified for measuring Latency in [RFC2544], e.g. measuring the latency of a single test frame in a traffic flow, is unsuitable for DCB benchmarking. 5.1.5.1. Forwarding Delay and Latency Multiple methods exist for measuring the time it takes a test frame to be forwarded by a DUT. However, both of the methods discussed in [RFC1242] are unsuitable for testing DCB devices, as many DCB devices alternate between both "store and forwarding" and "bit forwarding" behavior depending upon their queue congestion state. Hence, the only recommended method for measuring the time it takes a DUT to forward a test frame is "Forwarding Delay" as described in [RFC4689]. 5.1.6. Frame Sizes 5.1.6.1. Ethernet The recommended frame sizes for Ethernet testing are 64, 128, 256, 512, 1024, 1280, 1518, 4096, 8192, and 9216 as per [RFC5180]. Note that this frame size includes the Ethernet CRC. This frame size MUST include a VLAN header, since [802.1Qbb] requires the use of VLAN priority code points to distinguish different Classification Groups. 5.1.6.1.1. Fibre Channel over Ethernet FCoE test traffic introduces a number of frame size constraints that make the default frame sizes specified in [RFC5180] unusable: 1. FCoE frames contain an encapsulated Fibre Channel frame. Due to the method of encapsulation used, all FCoE frames MUST be a multiple of 4 bytes. See [RFC3643]. 2. Test tools may need to include a test payload in addition to the encapsulated Fibre Channel frame in order to meet the requirements specified in Section 5.1.4. 3. The maximum supported frame size for FCoE is 2176 bytes. Due to these constraints, the recommended frame sizes for FCoE testing are 128, 256, 512, 1024, 1280, 1520, 2176, and the smallest Player & Newman Expires February 15, 2010 [Page 10] Internet-Draft DCB Benchmarking August 2009 FCoE frame size supported by the test tool. This frame size MUST include a VLAN header, since [802.1Qbb] requires the use of VLAN priority code points to distinguish different Classification Groups. Player & Newman Expires February 15, 2010 [Page 11] Internet-Draft DCB Benchmarking August 2009 6. Benchmarking Tests 6.1. Queueput 6.1.1. Objective To determine the Queueput for one or more Traffic Classifications for a DUT using priority-based flow control. 6.1.2. Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Each Classification Group MUST be listed. For each classification group, the following parameters MUST be specified: Codepoint - For DCB tests, the codepoint is the VLAN priority. Frame Size - The frame size includes both the CRC and VLAN header. See Section 5.1.6x for recommended frame sizes. Burst Size - The burst size specifies the number of frames transmitted with the minimum legal IFG before pausing. A burst size of 1 specifies a constant load. Intended Vector - The intended vector SHOULD specify the intended rate of test traffic specified as a percentage of port load. Conformance Vector - The conformance vector is optional, but must be defined if used. Priority-based flow control - PFC mechanisms MUST be enabled. Background Traffic - Background traffic MAY be present. 6.1.3. Procedure A search algorithm is used to determine the Queueput for each Classification Group. If Queue Congestion is detected for a Classification Group during a trial, then the Intended Vector for the Classification Group MUST be reduced for the subsequent trial. If a Conformance Vector is specified for the test and nonconformant frames are received during a trial, then the Intended Vector SHOULD be reduced for the subsequent trial. The algorithm MUST adjust the Intended Vector for each Classification Group. The search algorithms for each Classification Group MAY be run in parallel. The test Player & Newman Expires February 15, 2010 [Page 12] Internet-Draft DCB Benchmarking August 2009 continues until all Classification Groups in the test have converged on a discrete Queueput value. 6.1.4. Measurements The Queueput for each Classification SHOULD be reported in either frames or bits per second. If a Conformance Vector is specified for a Classification Group, the percentage of conformant frames SHOULD be reported. The number of PFC pause frames for each code point in the Codepoint Set MUST be reported. 6.1.5. Reporting Format TBD 6.2. Maximum Forwarding Rate 6.2.1. Objective To determine the maximum forwarding rate of one or more PFC queues on a PFC capable DUT. 6.2.2. Setup Parameters Maximum Forwarding Rate is conceptually similar to the measurement in [RFC2285] but works on a per-Classification basis in a DCB context. The following parameters MUST be defined. Each variable is configured with the following considerations. Each Classification Group MUST be listed. For each classification group, the following parameters MUST be specified: Codepoint - For DCB tests, the codepoint is the VLAN priority. Frame Size - The frame size includes both the CRC and VLAN header. See Section 5.1.6 for recommended frame sizes. Burst Size - The burst size specifies the number of frames transmitted with the minimum legal IFG before pausing. A burst size of 1 specifies a constant load. Intended Vector - The intended vector includes the intended rate of test traffic specified as a percentage of port load. Player & Newman Expires February 15, 2010 [Page 13] Internet-Draft DCB Benchmarking August 2009 Conformance Vector - The conformance vector includes all QoS bounds required for this traffic group. Priority-based flow control - priority-based flow control mechanisms MAY be enabled or disabled. Background Traffic - Background traffic MAY be present. 6.2.3. Procedure The tester should iterate across all configured permutations of frame size, burst size, and Intended Vector for all Classification Groups. 6.2.4. Measurements The forwarding rate of each Classification Group SHOULD be reported as the number of test frames per second or bits per second the DUT correctly forwards to the proper egress port. The maximum forwarding rate for each Classification Group MUST be reported as the highest recorded forwarding rate from the set of all iterations. Both the Intended and Offered Vector of each Classification Group MUST be reported. The number of PFC Pause frames received for each Classification MUST be reported. If a Conformance Vector is specified for a Classification Group, the percentage of conformant frames SHOULD be reported. 6.2.5. Reporting Format TBD 6.3. Back-off 6.3.1. Objective To determine the delta between the maximum forwarding rate of a DUT and the point where the DUT ceases to use PFC to manage priority queues. 6.3.2. Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Player & Newman Expires February 15, 2010 [Page 14] Internet-Draft DCB Benchmarking August 2009 Each Classification Group MUST be listed. For each classification group, the following parameters MUST be specified: Codepoint - For DCB tests, the codepoint is the VLAN priority. Frame Size - The frame size includes both the CRC and VLAN header. See Section 5.1.6 for recommended frame sizes. Burst Size - The burst size specifies the number of frames transmitted with the minimum legal IFG before pausing. A burst size of 1 specifies a constant load. Intended Vector - The intended vector includes the intended rate of test traffic specified as a percentage of port load. Conformance Vector - The conformance vector includes all QoS bounds required for this traffic group. Priority-based flow control - priority-based flow control mechanisms MUST be enabled. Backoff method - The recommended backoff method is to reduce the aggregate traffic load by a fixed amount while still maintaining a fixed load ratio between all Classification Groups. 6.3.3. Procedure The initial trial SHOULD begin with an Intended Load equal or greater than the Maximum Forwarding Rate of the DUT/SUT. For each subsequent trial, the aggregate load is reduced until the DUT is observed to complete a trial without activating any Congestion Management methods. 6.3.4. Measurements The Intended and Offered Vector for each Classification Group MUST be reported. The number of PFC Pause frames received for each Classification Group MUST be reported. If a Conformance Vector is specified for a Classification Group, the percentage of conformant frames SHOULD be reported. 6.3.5. Reporting Format TBD Player & Newman Expires February 15, 2010 [Page 15] Internet-Draft DCB Benchmarking August 2009 6.4. Back-to-Back 6.4.1. Objective To determine the maximum burst size a DUT handle on one or more PFC queues. 6.4.2. Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations Each Classification Group MUST be listed. For each classification group, the following parameters MUST be specified: Codepoint - For DCB tests, the codepoint is the VLAN priority. Frame Size - The frame size includes both the CRC and VLAN header. See Section 5.1.6 for recommended frame sizes. Burst Size - The burst size specifies the number of frames transmitted with the minimum legal IFG before pausing. A burst size of 1 specifies a constant load. Intended Vector - The intended vector includes the intended rate of test traffic specified as a percentage of port load. Conformance Vector - The conformance vector includes all QoS bounds required for this traffic group. Priority-based flow control - priority-based flow control mechanisms MAY be enabled or disabled. The sum of all Intended Vectors on a transmitting port SHOULD equal the Maximum Offered Load (MOL) of that port. 6.4.3. Procedure A search algorithm is used to determine the maximum duration in seconds for which the configured Classification Profile can be forwarded by the DUT without active Congestion Management. If Congestion Management is detected during an iteration, then the duration MUST be reduced for the next iteration. 6.4.4. Measurements The Intended and Offered Vector for each Classification Group MUST be reported. Player & Newman Expires February 15, 2010 [Page 16] Internet-Draft DCB Benchmarking August 2009 The number of PFC Pause frames received for each Classification Group MUST be reported. If a Conformance Vector is specified for a Classification Group, the percentage of conformant frames SHOULD be reported. 6.4.5. Reporting Format TBD Player & Newman Expires February 15, 2010 [Page 17] Internet-Draft DCB Benchmarking August 2009 7. Security Considerations A broken PFC implementation could lead to hypothetical denial-of- service attacks where a DUT ignores PFC pause frames and incorrectly forwards traffic as a result, or conversely throttles traffic even though no congestion condition exists. However, the test networks described in Benchmarking Methodology Working Group documents generally SHOULD NOT be reachable by anyone other than the tester(s). The security implications of such incorrect PFC handling are beyond the scope of this performance-measurement document, which is intended for testing purposes in controlled lab conditions. Player & Newman Expires February 15, 2010 [Page 18] Internet-Draft DCB Benchmarking August 2009 8. IANA Considerations Testers SHOULD use network addresses assigned by IANA for the purpose of testing networks. Player & Newman Expires February 15, 2010 [Page 19] Internet-Draft DCB Benchmarking August 2009 9. Normative References [802.1Qbb] Institute of Electrical and Electronics Engineers, Inc., "802.1Qbb - Priority-based Flow Control", 2009. [RFC1242] Bradner, S., "Benchmarking terminology for network interconnection devices", RFC 1242, July 1991. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, February 1998. [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999. [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology for LAN Switching Devices", RFC 2889, August 2000. [RFC3643] Weber, R., Rajagopal, M., Travostino, F., O'Donnell, M., Monia, C., and M. Merhar, "Fibre Channel (FC) Frame Encapsulation", RFC 3643, December 2003. [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast Benchmarking", RFC 3918, October 2004. [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, "Terminology for Benchmarking Network-layer Traffic Control Mechanisms", RFC 4689, October 2006. [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. Dugatkin, "IPv6 Benchmarking Methodology for Network Interconnect Devices", RFC 5180, May 2008. Player & Newman Expires February 15, 2010 [Page 20] Internet-Draft DCB Benchmarking August 2009 Appendix A. Acknowledgements Player & Newman Expires February 15, 2010 [Page 21] Internet-Draft DCB Benchmarking August 2009 Authors' Addresses Timmons C. Player Spirent Communications Email: timmons.player@spirent.com David Newman Network Test Email: dnewman@networktest.com Player & Newman Expires February 15, 2010 [Page 22]