Table Of Content

Source: Fibre Channel for SANs 1 Chapter Fibre Channel and Storage Area Networks Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 2 Chapter 1 Introduction Fibre Channel technology is over a decade old. How successful has it been? Here is an illustration. The first edition of this book included a section called “The Unification of LAN and Channel technologies,” which described how Fibre Channel would be part of a trend towards convergence between LANs and channels. LANs (Local Area Networks) are used for computer-to-computer communications, and channels are high-efficiency, high-performance links between computers and their long-term storage devices (disk and tape drives), and other I/O devices. Since then, the prediction has come true, in three quite different ways. • Most important has been the introduction and widespread use of the term “StorageAreaNetwork,” or SAN, describing a network which is highly optimized for transporting traffic between servers and storage devices. • At the physical layer, the LAN and Fibre Channel technologies have become nearly identical — Gigabit Ethernet and Fibre Channel share com- mon signaling and data encoding mechanisms, and the future 10 Gb/s Ethernet and Fibre Channel are expected to share nearly the same data rate. • The management methods for Fibre Channel SANs have steadily approached the traditional methods used for LAN management, although the current level of management effort required for Fibre Channel SANs is still higher than for LANs. Interestingly, however, although the LAN and SAN types of computer data communications have converged at a technology level, they have so far stayed quite different in how they are used and how they are managed. That is, systems are usually built with the SAN storage traffic separated on separate networks from the LAN traffic, so that the management, topologies, and provisioning of each network can be optimized for the types of traffic tra- versing them. The trends that originally motivated the creation of Fibre Channel have continued or accelerated. The speed of processors, the capacities of memory, disks, and tapes, and the use of switched communications networks have all been doubling every 18 to 24 months, and the doubling period has in many cases even been steadily shortening slightly. However, the rate of I/O improvement has been much slower, so that devices are even more I/O lim- ited. The continuing observation is that computers usually appear nearly instantaneous, except when doing I/O (e.g., downloading web pages), or managing stored data (e.g., backing up file systems). Fibre Channel, and Storage Area Networks, are focused at (a) optimizing the movement of data between server and storage systems, and (b) managing the data and the access to the data, so that communications are optimized as Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 3 Fibre Channel and Storage Area Networks much as possible, while continuously and reliably providing access to data, for whoever needs it. Fibre Channel Features Following is a list of the major features that Fibre Channel provides: • Unification of networking and I/O channel data communications: This was described in detail above, and allows storage to be decoupled from servers and managed separately. Similarly, many servers can directly access the data as if it were their own, as long as they are coordinated to manage it coherently. • Bandwidth: The base definition of Fibre Channel provides better than 100 MBps for I/O and communications on current architectures, with speeds defined up to 4 times this rate, for implementation as market and applications dictate. • Inexpensive implementation: Fibre Channel uses an 8B/10B encoding for all data transmission, which, by limiting low-frequency components, allows design of AC-coupled gigabit receivers using inexpensive CMOS VLSI technology • Low overhead: The very low 10-12 bit error rate achievable using a combi- nation of reliable hardware and 8B/10B encoding allows very low extra overhead in the protocol, providing efficient usage of the transmission bandwidth and saving effort in implementation of low-level error recovery mechanisms. • Low-level control: Local operations depend very little on global information. This means, for example, that the actions that one Port takes are only minimally affected by actions taking place on other Ports, and that individ- ual computers need to maintain very little information about the rest of the network. This feature minimizes the amount of work to do at the higher levels. • For example, hardware-controlled flow control alleviates the host processors from the burden of managing much of the flow control overhead. • Similarly, the low-level hardware does sophisticated error detection and deletion, so that it can assure delivery of data intact or not at all. Upper layer protocols don’t have to do as much error detection, and can be more efficient. • Flexible topology: Physical connection topologies are defined for (1) point-to-point links, (2) shared-media loop topologies, and (3) packet- switching network topologies. Any of these can be built using the same Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 4 Chapter 1 hardware, allowing users to match physical topology to the required con- nectivity characteristics. • Distance: 50 m in a room simplifies wiring, more important is 10 km, which allows remote copy without WAN infrastructure. Consider a high performance disk drive attached to a computer over an optical fiber. The access time for the disk drive (to rotate the disk and move the head over the data) would be roughly 5 ms. The speed of light in optical fiber is about 124 mi/ms. This means that the time to reach an optically connected disk drive located a mile away would be only 0.008 ms more than the time to reach a disk drive in the same enclosure. • Availability: More capability to attach to multiple servers allows the data to be accessed through many paths, which enhances availability in case one of those paths fails. • Flexible transmission service: Mechanisms are defined for multiple Classes of services, including (1) dedicated bandwidth between Port pairs at the full hardware capacity, (2) multiplexed transmission with multiple other source or destination Ports, with acknowledgment of reception, and (3) best-effort multiplexed datagram transmission without acknowledgment, for more efficient transmission in environments where error recovery is handled at a higher level, (4) dedicated connections with configurable quality of service guarantees on transmission bandwidth and latency, and (5) reliable multicast, with a dedicated connection at the full hardware capacity. • Standard protocol mappings: Fibre Channel can operate as a data transport mechanism for multiple Upper Level Protocols, with mappings defined for IP, SCSI-3, IPI-3 Disk, IPI-3 Tape, HIPPI, the Single Byte Channel Com- mand set for ESCON, the AAL5 mapping of ATM for computer data, and VIA or Virtual Interface Architecture. The most commonly used of these currently are the mapping to SCSI-3, which is termed “FCP,” and the mapping to ESCON, which is termed either “FICON,” or “SBCON,” depend- ing on context. • Wide industry support: Most major computer, disk drive, and adapter man- ufacturers are currently developing hardware and/or software components based on the Fibre Channel ANSI standard. These improvements to traditional channels don’t actually provide much real benefit when a single server is used to process the data on a single storage device. However, when multiple servers act together (for better reliabil- ity, or higher throughput, or better pipelining, etc.) to work with the data on multiple storage devices of different types, then the advantages of Fibre Channel can become very important. Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 5 Fibre Channel and Storage Area Networks Storage Area Networks What is a Storage Area Network, and how is it different from the various other types of networks that are built? Here is a definition of a Storage Area Network, from one of the leaders in the industry: A Storage Area Network (SAN) is a dedicated, centrally managed, secure information infrastructure, which enables any-to-any interconnection of servers and storage systems. This definition is unfortunately not particularly instructive as to, for example, the difference between SANs and LANs, or MANs, or even WANs, all of which, in some applications, could fit this description. The difference between SANs and other types of networks can perhaps best be understood by considering the difference between the storage and networking ports on a desktop computer. Every computer has access to some kind of long-term storage, and almost every computer has access to some way of communicating with other computers. The storage interface is highly optimized, tightly controlled (in laptops and most desktop machines, it may not even be visible outside the box), and not shared with any other computers — which helps make it highly predictable, efficient, and fast. Network interfaces, on the other hand, are much slower, less efficient (you have to wait for them), and have higher overhead, but they allow access to any other machine that it knows how to communicate with. Storage Area Networks are built to incorporate the best of both storage and networking interfaces: fast, efficient communications, optimized for efficient movement of large amounts of data, but with access to a wide range of other servers and storage devices on the network. The primary difference then between a Storage Area Network and the other types of networks mentioned is that, in a SAN, communication within the network is well-managed, very well-controlled, and predictable. There- fore, each entity on the network can almost operate is if it has sole access to whichever partner on the network that it is currently communicating with. A primary reason for this has been the idea of decoupling the servers from their storage, and allowing multiple servers to access the same data at the same time. The key here is that client systems often access their through servers, which assure consistency, security, and authorization for the data access. Clients, however, don’t particularly care which server is used to access the data, and the data is the same no matter which server is accessing it. This three-tiered system of clients displaying the data, servers processing and managing the data, and storage subsystems holding the data, is tied together with networks — LANs and SANs — between each layer. Fibre Channel overlaps very little with Ethernet, except in very specific applications. For general-purpose communications, Ethernet is very difficult Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 6 Chapter 1 to compete with (particularly since the Ethernet community tends to adopt the best networking innovations every time there is a new generation, which is regularly). Fibre Channel does, however, overlap very closely with the storage technologies such as IDE and SCSI. In fact, to a file system or higher-level device, Fibre Channel may appear almost exactly like SCSI — the SCSI command set is transported across a Fibre Channel link, just as it would be across a SCSI bus. The preceding picture is generally valid for on mid-range machines. On high-end machines, the networking interface is usually still Ethernet (although Token Ring, FDDI, HiPPI, and others have all been important), but the storage interface has, for the last 10 years or so, been a channel protocol. The primary one in the early ’90s was called ESCON, for Enterprise System Connections. ESCON was the first real SAN, since it allowed multiple servers to access multiple storage units through a high-performance, switched fabric. In fact, currently the ESCON protocols are still transmitted over a high-performance, switched fabric, but now the fabric is Fibre Chan- nel, and the name has changed to FICON or SBCON. SAN topologies A typical topology for a large-scale system using both a Fibre Channel- based Storage Area Network and a Local Area Network is shown in Figure 1.1. This configuration allows a number of advantages, vs. a system with storage devices tightly integrated with each separate server. • Networked Access: All servers have direct access to all disk and tape arrays through the SAN, once authorization has been established at the network and the data level. • Storage Consolidation: Since the client, server, and storage units can be scaled separately, and storage units can be shared, fewer units are neces- sary. This is especially important for expensive, large tape libraries. • Remote Mirroring and Archiving: Since the SAN links may be up to 10 km. long, disk and tape drives can be remotely located, for disaster recovery. • LAN-free backup. The servers can move the data between disk and tape arrays over the SAN — so the LAN between server and clients is not impacted by the backups, and is always available. • Server-free backup. In the ideal case, the disk array and the tape array have enough intelligence to let the servers command 3rd-party transfers, so that, for example, data would flow directly between a disk array and tape library Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 7 Fibre Channel and Storage Area Networks Desktops Figure 1.1 or Example of an Enterprise Laptops or Service Provider To SAN+LAN Topology WAN Router LAN Switch Servers with local storage SAN Switch SAN Switch Disk Array Tape Library Storage Devices Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 8 Chapter 1 across the SAN, without loading any servers. These capabilities are getting steadily more important. In 1999, roughly 3/4 of the storage sold in the world was attached directly to servers, while the remaining part was attached directly to the network. In 2003, over 3/4 of storage is expected to be directly attached to the networks, either as SAN or NAS storage. SANs, LANs, and NAS A major issue in the design of complex installations such as this involves the set of difference between LANs and SANs, particularly, since there are a large number of storage devices, termed “Network Attached Storage,” that attach to Ethernet LANs. In general, the fact is that SAN traffic is faster and more efficient than LAN traffic. Getting over 80% throughput on SAN links is expected, while getting over 30% on a sustained basis on LAN links is doing well. More importantly, the processor overhead for communications is generally much higher on LANs, than in SANs. Some estimates are that the processor overhead for TCP/IP on a LAN is 1,000 MIPS to receive data at 1 Gb/s, and that the processor overhead running TCP/IP over Ethernet is 30 times higher than running the same data rate over Fibre Channel. The 30X performance difference is quite amazing — what could possibly cause two networks with the same line speed to use 30X difference in processor protocol-processing overhead? The following sections attempt to explain this in some detail. A caution on this section. Many of these factors (1) are extremely dependent on implementation, and (2) are changing extremely quickly — so don’t expect them to be always true everywhere. The main reason for listing them here is to help people understand how to optimize design of networks and network interfaces. LANs vs. SANs: Differences in Network Design Some of the efficiency advantages of Fibre Channel compared to Ethernet relate directly to the design of the network. In an environment of steady innovation, any real design advantages get quickly adopted in all following- generation designs, so these are only short-term advantages. • Low-level (hardware-based) link-level and end-to-end flow control, so the higher levels don’t have to manage flow control and congestion control. High-level flow control and congestion control (e.g., the TCP window Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 9 Fibre Channel and Storage Area Networks mechanism, slow start and congestion avoidance) can require significant overhead, especially on heavily-loaded networks. • Switch-based transmission (vs. shared medium), so the quality of service for a particular connection can be higher. • Upper-level protocol information defined in the network-level headers, so low-level hardware can effectively assist higher-level protocol processing. Again, the network layer for Fibre Channel is not much different than modern Ethernet on a switched fabric (i.e., not shared medium), with link- level backpressure flow control. There are some advantages to the Fibre Channel network vs. Gigabit Ethernet, but not a 30X difference. LANs vs. SANs: Differences in Protocol Design The more important advantages in SAN efficiency vs. LAN efficiency and performance relate to the higher levels of protocol design, and have to do with the fact that LANs are, in general, accessed through a TCP/IP (or UDP/IP) protocol stack, where SANs are accessed through a simpler SCSI protocol stack with less overhead on the host processor. This include the following factors. • Lower-lever error checking. The channels deliver the data to the server intact, or not at all (data corruption, or pulled cable) — so the processors do less checksum calculation or validation of header fields, for example. • Predictable network performance • Ordered transmission — assume no re-ordering of traffic on the network, so the extra overhead associated with checking for correct delivery order, and resource allocation to compensate if you don’t have it, are gone. • Well-defined network round-trip times, so that the protocol doesn’t have to include code to handle the “did the packet get lost, or is it just badly delayed?” problem. • Request/Response network — the server makes requests to the disk sub- system for reads or writes, so all incoming packets to the server are expected packets. This means: • Less header parsing and less handling of special cases, since all packets coming in are expected, and resources for dealing with them have been pre-allocated. • Less overhead for flow control — no need to allocate buffer space or do buffer management processing for traffic which may or may not come in. • Message-based transport: TCP is a sockets stream protocol, where SCSI works in command or data blocks, or messages, with memory space pre- Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website. Fibre Channel and Storage Area Networks 10 Chapter 1 allocated, so less buffer management, and less data copying, are required in many cases. • Higher granularity of transfers — Ethernet adapters typically work at the level of Ethernet packets, with all higher-level segmentation and reassembly into IP datagrams, or TCP-level sockets, requires host processor inter- vention. Fibre Channel adapters typically do reassembly of Frames into Sequences, and deliver the full Sequence to the ULP for processing by the host processor. This means, for example, that there may be fewer processor interrupts, and less context switching. • Real address operations — SCSI protocols work in the kernel, so there’s no switching from user context to kernel context, and real addresses can be used in all the operations, so may be less translation between virtual and physical addresses. Network-attached Storage (NAS) and Storage Area Networks (SAN) An area that is closely tied to this difference between LANs and SANs is the difference between NAS and SANs. It is sometimes difficult to be sure of the function difference between the two, partly because they nearly share an acronym, and partly because they both allow networked access to stored data. However, they really are quite different from each other, both in func- tionality and how they are used. Part of the difference between Network-attached Storage, and a Storage- Area Network has to do with the network and protocol stack used. Network attached storage emphasizes the network: Ethernet networks and TCP/IP or UDP/IP protocol stacks), where Storage Area Networks use Fibre Channel and a SCSI protocol stack. The hardware difference is less important than the higher layer differences, however, particularly if both networks operate at nearly the same speed and topology. A more important key to the difference between NAS and SAN is the dis- tinction in which kind of traffic crosses the network. In NAS, the traffic crossing the network is high-level requests and responses for files, independ- ent of how they are arranged on disks. In SAN, however, the traffic is requests and responses for blocks of data at specific locations on specific disks. The difference here is that NAS operates above the file system level, where SANs operate below the file system level, at the data block level. A network-attached storage device is a dedicated file server which holds files, and exports to the clients a picture of a file system. The clients request reads or writes to files, and the network-attached storage device does the Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com) Copyright © 2004 The McGraw-Hill Companies. All rights reserved. Any use is subject to the Terms of Use as given at the website.