A comparison of the Fibre Channel and SerialExpress standards UW Section Executive Summary The need to transport high volumes of data at Gbit speeds has led to the invention of new connection technologies. While Gbit ethernet has received a lot of press recently as the solution to this problem, several other technologies have emerged as potential contenders. One of these is Fibre Channel. Fibre Channel was established as a family of ANSI standards in 1995. Hardware to support this standard has been available since that time. A dark horse contender in the race to claim the high-speed connection race is the SerialExpress technology. The definition of the SerialExpress standard was started in 1996 in response to perceived deficiencies in the IEEE1394 (FireWire) standard. It will take several years before the SerialExpress technology is ready for use. It is being described here because it comes at the problem of transporting data at high speeds from a different perspective. Overview Fibre Channel is based on a family of ANSI standards which define a channel based interconnect architecture. The Fibre Channel standards are organized in a five-layer OSI-like stack. The first layer, FC-0 defines the physical media and transmission rates. The next layer, FC-1 defines byte encoding and decoding. This layer is based on the IBM 8B/10B 10-bit transmission coding scheme. FC-2 describes the framing protocol, service classes, and flow control features of Fibre Channel. These first three layers form the Fibre Channel physical standard, FC-PH. The next layer, FC-3 defines special service features of Fibre Channel. These features include descriptions of how data will be striped when using Fibre Channel for accessing disk arrays as well as special broadcast and multicast functions to be used when interfacing with a video server. The top Fibre Channel layer, FC-4, describes multiple service interconnects. This layer provides interfaces to legacy protocols. Among the interfaces defined at this level are IPI, SCSI, HIPPI, 802.2, IP, ATM, and Escon. The SerialExpress draft standard (IEEE1394.2) describes a set of signals and protocols which can be used to communicate at higher bandwidths and longer distances than is possible with the IEEE1394 (FireWire) Serial Bus. SerialExpress protocols have evolved from the Serial Bus protocols with refinements and changes derived from experience and the need to support higher bandwidths over longer distances. In addition, SerialExpress has borrowed heavily from the Scaleable Coherent Interface (SCI) standard. While Fibre Channel represents a more traditional approach to a network interconnect standard, SerialExpress is based on a shared memory bus architecture. Key Issues The next section compares the Fibre Channel and the SerialExpress standards. The comparison loosely follows an OSI-like layered structure. Initially both standards are compared at the transmission media layer. Each standard claims a 1 Gbit performance with 2 Gbit and 4 Gbit possible in the future. Another area for comparison at this level is the maximum distance between nodes. Finally support for standard cabling technologies is considered. The next layer deals with encoding bits for transmission. Both standards use IBM's proprietary 8B/10B technique. The IBM coding approach is briefly described. The third layer consists of a number of areas for comparison between the two standards. Most of the comparison between SerialExpress and Fibre Channel is performed at this level. The features considered here include the standard's frame size, the addressing scheme used, standard's supported topologies, and the classes of supported services. Finally the comparison concludes with a look at how each of these standards deal with upper layer legacy protocols. Analysis The Fibre Channel FC-0 defines support for a wide range of physical cabling options. Fibre Channel can use single mode fiber optic cable, multi-mode 50 and 62.5 micrometer fiber, video coax, miniature coax, and shielded twisted pair. The distance covered between Fibre Channel nodes can range from 10 meters for 1 Gbit miniature coax up to 20 km for single mode 1 Gbit longwave laser fiber. The FC-0 specification was defined with flexibility as a goal. This allows systems integrators to choose from a wide range of selections for speed and distance. Like Fibre Channel, a goal of SerialExpress was to be transmission-media independent. The draft standard claims that SerialExpress can mix and match fiber and copper media. While SerialExpress can use either fibre or copper, it requires special cables which has either two links within one cable or one link with directional sensitive couplers. Copper SerialExpress cables also include two power lines. This allows low cost devices to receive their power from the other node at the other end of the cable. When copper cables are connected to fibre cables, the power lines terminate at the coupler and provide power to the fiber optic transducers. The standard indicates a maximum distance of 5 meters between connections for electrical interfaces. Change requests in the appendix J of the draft indicate that there is an agreement to extend the standard to support distances greater than 25 meters for fiber optic interfaces. From this discussion, it is clear that Fibre Channel provides the most flexibility when choosing the transmission media. SerialExpress requires special cabling which provides a limited distance of only 5 meters for copper links with up to 25 meters for fiber optic links. Fibre Channel has the advantage with a distance between nodes of up to a maximum length of 20 km. Both standards use IBM's 8B/10B encoding scheme to code bits for transmission. This approach encodes 8-bit characters into 10 bits. To do this, the first 5 bits of the 8-bit character are coded into 6 bits. The remaining 3 bits are coded into a 4-bit code. The 6 bit and 4 bit codes are then combined into 10 bits. The coding insures a maximum run of 5 (i.e. a run a 5 ones or 5 zeros). This coding solves two potential coding problems. It provides a sufficient number of transitions to aid with clock synchronization and helps maintain DC balance on the line. The FC-2 signaling layer defines the framing for data transmission. The general format for a FC-2 frame is 4 bytes for a Start of Frame ordered set followed by 24 bytes of header information, followed by a payload with up to 2112 bytes of data. The payload bytes are followed by a CRC-32 error check. The frame is terminated by 4 bytes for the End of Frame ordered set. The header may be followed by 64 bytes of optional header information. In which case the data payload is limited to a maximum of 2048 bytes. The frame header contains the source and destination identifiers, control information, sequence information, and a frame type indicator. SerialExpress packets have a 16 byte header with a payload of 64 bytes of data. The standard also defines an option which allows a packet to carry 256 bytes of data. This was included to provide backward compatibility with both IEEE1394 and SCI. However, in cases where the SerialExpress hardware does not support the optional size, the larger sized packets will not be transported and will cause a type error to be generated. All packets include a target identifier field, a codes field and a command-code field. Other information included in the header is dependent on the type of packet being transmitted. The choice of packet size contributes to the amount of latency when transporting packets. SerialExpress uses smaller sized packets to reduce latency and to simplify the design of node hardware. Fibre Channel uses a more traditional packet size which potentially leads to more expense in the Fabric hardware and a higher latency. Fibre Channel assigns to each N_port a 24-bit native identifier unique within the address domain of a Fabric. Each N_port may also have one or more alias address identifiers which may be shared across multiple N_ports. Alias addressing may be used to implement a Hunt Group. Each packet has a source node identifier (the address identifier of the originating port) and a destination node identifier. At system initialization, the Fabric may assign identifiers to each N_port or F_port in the Fabric's address domain. Other methods for assigning port address identifiers may be used but these alternative methods are not described by the FC_PH standard. Each SerialExpress ringlet is initialized independently and each attached node is assigned a unique hardId location on busNumber 1023. The hardId consists of a 10-bit 1023 value concatenated with a 6-bit slotId value. The slotId is the hop distance of the node from the scrubber. On bus reset, the hardId may be assigned a different value. Each node also has a softId 16-bit value which is assigned to the node when the SerialExpress Fabric is initialized. This value is retained on a bus reset. For directed requests, the 16-bit node identifier is combined with a 48-bit address offset which directs the request to a location on the node. Fibre Channel links are connected to ports. Ports can be either N-ports (node ports) or F-ports (fabric switch ports). Each port has a receiver and a transmitter, Fibre Channel is full duplex with the receiver of one port connected to the transmitter of the connecting port. This allows concurrent bi-directional communication between ports. FC-2 defines the signaling between ports and other traffic management functions, including flow control, link management, buffer memory management, and error detection and correction. To support an arbitrated loop topology, the Fibre Channel also defines NL-ports (node loop ports) and FL-ports (fabric loop ports). The Fibre Channel arbitrated loop topology is described in the FC-AL draft standard. Like Fibre Channel, SerialExpress nodes are connected to each other by full duplex bi-directional ports. SerialExpress nodes may have up to 3 ports. Each node also includes an attach port for connecting to the node's device. Multi-port nodes have an internal switch between the ports and the attach port. The combination of bi-direction ports and the internal switch allows a node to disconnect redundant and malfunctioning nodes during system initialization and to provide automatic "loop-back" of data from a disconnected port. The Fibre Channel supports three distinct topologies; point-to-point, fabric, and loop. The point-to-point topology is non-blocking. Each N_port may transmit frames to the other at any time within the constraints of the protocols implemented on the N_ports. This provides instantaneous access to the other N_port. But, this has the potential of under-utilizing the capabilities of the N_port. When a link in the point-to-point topology fails, the communication between that link's pair of Ports stops. Other ports in the point-to-point topology can continue to communicate. For the Fabric topology, Fibre Channel switches are interconnected to form the fabric. Fibre Channel nodes are connected to the fabric by F_nodes. The Fabric topology permits multiple paths to exist between any two F_ports in the fabric. The Fabric topology is more robust than the point-to-point topology, but there is still a single point of failure for any nodes connected to the Fabric by only one N_port. The Fabric topology allows N_ports to share the available bandwidth but this sharing introduces resource contention. Because of this contention, the Fabric requires a management function to control the movement of packets through the fabric. The arbitrated loop topology is formed by connecting Fibre Channel nodes in a loop using L_ports. The L_port is constructed in such a way as to only accept packets addressed to a node and to pass all other packets on through the port. For two nodes to communicate over the loop, they first establish a connection over the loops shared link. While these two nodes are communicating all other nodes are kept waiting. The connection is only released after the two nodes are done. The method for arbitrating which nodes gain access to the loop is designed to be fair. To do this a window is formed from all of the nodes wishing to use the loop. All nodes in this window are guaranteed use of the loop before the next window of waiting nodes is established. This protocol does not keep a node from participating more than once in any one window period. While any number of L_ports can be connected to a loop, only 127 ports can be active at any one time. Arbitrated loops can also be connected to the Fabric through an FL_port. Fibre Channel loops fail if any link within the loop fails. The SerialExpress bus implements a ringlet topology with up to 63 nodes connected together on one ringlet. Up to 1023 ringlets can be connected to each other using bus bridges. As noted above, each SerialExpress node can have up to 3 ports. This allows nodes to be connected with redundant connections or in a network like topology. During bus initialization or on a bus reset, redundant or malfunctioning links are disabled to form the ringlet. Also during this period, one node on the ringlet is selected using a node-identifier-precedence based voting protocol. The selected node, called the scrubber, is responsible for a variety of ringlet maintenance functions. After initialization, the scrubber initiates communication on the ringlet by transmitting a series of idle packets. As each node needs access to the ringlet to send a packet, it replaces an idle packet with a packet it wishes to send. As packets are removed (striped) from the ringlet, they are replaced by idle packets. To help arbitrate conflicts on the ringlet , each node has a pass-through buffer and a buffer for packets originating at the node. The node implements a set of rules to determine which buffer is sent out next. The details are not important, what is important is that each node detects idle periods on the ringlet and aggressively transmits during these lulls. This approach for sharing the ringlet for communication and the limited sized of the SerialExpress packets help reduce the latency of the ringlet architecture. The SerialExpress draft standard suggests that ringlets can be connected through some form of switched network topology such as a central switch closet. The standard does not supply specifics on how this would be accomplished. From a perspective of number of topologies defined in each of the standards, Fibre Channel seems to be the clear winner. But, when comparing the ringlet against the arbitrated loop, the ringlet has several advantages. Theoretically, the ringlet has a lower latency due to several factors; the small size of its packets, the ability to effectively share the ringlet, and the distributed arbitration of conflicts. The ringlet topology is also more robust because it can automatically recover from a failed link by doing a bus reset to reconfigure the ringlet topology. The Fibre Channel describes three classes of service for N_ports and the Fabric. The class 1 service provides a dedicated connection between two N_ports. This connection is established when an N_port requests a connection to another N_port using a login protocol. The destination N_port sends an acknowledgment to establish the connection. The connection is maintained and guaranteed by the Fabric until one of the communicating N_ports explicitly requests that the connection be removed. The class 1 service follows an end-to-end flow control protocol. This protocol requires that the destination node acknowledges receipt of a frame. The class 1 service also has an intermix mode which allows class 2 and class 3 frames to use any remaining bandwidth available after the class 1 connection is made. The class 2 service provides connectionless service with notification of non-delivery of frames between N_ports. This service is requested by a N_port on a frame by frame basis. The Fabric routes the frames to the destination N_port. This service uses both end-to-end and buffer-to-buffer flow control protocols. The buffer-to-buffer flow control protocol uses a request ready to signal that the Fabric is ready to accept the frame. The Fibre Channel class 3 service is a connectionless service without any notification of non-delivery, delivery, or end-to-end flow control between two N_ports. With this service the Fabric can drop frames without notifying the originating node. This service does use the bus-to-bus flow control protocol. SerialExpress has two types of delivery services; asynchronous transactions and isochronous streams. The asynchronous transactions use the commands read, write, lock, and move to provide flow control for the transfer of data between nodes. Isochronous streams use unconfirmed, fixed-rate transfers of data. The use of isochronous streams is prenegotiated at a higher level to insure that the cumulative demands on the ringlet don't exceed its capacity. This makes lower level flow control unnecessary. The delivery of data within each service can be directed to a single node or multicast to several nodes. The Fibre Channel provides more choices when selecting the class of service for transferring data between nodes. However, with this flexibility comes more complexity and overhead. As before, SerialExpress provides a simpler set of choices with the potential for reducing latency and keeping the design of the node hardware relatively simple. As noted earlier, the Fibre Channel FC-4 provides definitions for interfacing the lower level protocols to upper layer legacy protocols. In addition to these definitions, the physical layer standard describes protocols that can be used by the Fibre Channel Fabric when mapping external network addressing protocols to Fibre Channel node identifiers. The SerialExpress standard provides little or no direction on how to interface a SerialExpress Fabric to an external network. The standard does suggest a method for using ARP to map IP addresses to internal bus/node addresses. Conclusion By necessity, this paper has only briefly touched on just a few of the key features provided in these standards. As is often the case, the devil is in the details. The Fibre Channel standards provide a traditional network architecture with OSI-like layers where each layer builds on the services provided by lower layers. This separation of services provides Fibre Channel its flexibility and ability to integrate well with current network standards. It also leads to Fibre Channel's complexity. On the other hand, SerialExpress is very much a work in progress. It promises much in delivering data at high speeds and low latency by keeping its design simple. To accomplish this, SerialExpress does not implement a layered network architecture. Instead, it is a shared memory bus architecture with some features that allow it to adapt to the role of providing local area connections. The success or failure of SerialExpress to deliver on its promise depends on it becoming a standard while retaining compatibility with the FireWire standard. References. ANSI X3.230-1994, Fibre Channel Physical and Signaling Interface (FC-PH) Rev. 4.3. Draft version of ANSI standard. Available: http://www.amdahl.com/ext/CARP/FCA/FCPH-43.pdf. 1 June 1994 Fibre Channel Arbitrated Loop (FC-AL-2) REV 5.4. ANSI working draft. Available: http://www.network.com/ftp/FC/AL/fcal54d.pdf. 22 March 1997. Frymoyer, Edward M. Fibre Channel Fusion: Low Latency, High Speed. Data Communications on the web. Available: http://www.data.com/Tutorials/Fibre_Channel_Fusion.html. February 1995. Meggyesi, Zoltan. Fibre Channel Overiew. Research Institute for Particle and Nuclear Physics. Budapest, Hungary. Available: http://www1.cern.ch/HSI/fcs/spec/overview.htm. 15 August 1994. SerialExpress - A Scaleable Gigabit Extension to Serial Bus. Draft version .729. Microprocessor and Microcomputer Standards Subcommittee of the IEEE Computer Society. Available: ftp://ftp.SCIzzL.com/P1394.2/SerExp.729.pdf. 8 November 1996.