Flattening the Data Center Network - Bitpipe

Flattening the Data Center NetworkEthernet Fabric Revolutionizes Data Center Network Design

A Technology Validation Report

BrocadePrepared for

i

Flattening the Data Center Network

ii


Contents

About DeepStorage iii

The Bottom Line 1

Introduction 2

The Emergence of Ethernet Fabrics 3

Ethernet Fabrics 3

VM Mobility 4

Converged Networking 4

Our Modest Proposal 5

The Requirements 6

Brocade VCS Fabric Technology 7

Cost Analysis 8

Bandwidth and Failure Analysis 9

Management and Expansion 9

Applicability of the Model Design for Smaller and Larger Configurations 11

Conventional Spanning Tree Solution 11

Cost Analysis 12



Cisco Nexus 7000 with Cisco Nexus 2000 Fabric Extenders 13

Cost Analysis 14

About the Cisco Nexus 2000 Fabric Extender 15



Lab Testing 17

Our Test Configuration 20

vSphere Host Server Hardware 20

Virtual DVD Store Server Configuration 21

Conclusions 22

Appendix A – Network Configuration Bills of Materials 24

Brocade VCS Configuration 24

Cisco 6509-E/Cisco 4500-X Configuration 25

Notes on Cisco 6509-E/Cisco 4500-X Configuration 25

Cisco Nexus 7000/Cisco Nexus 2000 26

iii


About DeepStorage

DeepStorage, LLC is dedicated to revealing the deeper truth about storage, networking, and related data center technologies to help information technology professionals deliver superior services to their users and still get home at a reasonable hour.

DeepStorage Reports are based on our hands-on testing and more than 25 years of experience making technology work in the real world.

Our philosophy of real-world testing means we configure systems as we expect most customers will use them, thereby avoiding “Lab Queen” configurations designed to maximize benchmark performance.

This report was sponsored by Brocade. But we always retain final editorial control over our publications.

1


The Bottom Line

In recent years, the Ethernet market has evolved significantly. Part of that evolution has been the differentiation of data center networks from campus and metro Ethernet networks. New technologies have been developed specifically to address some of the unique challenges that each type of network presents.

Vendors, including Brocade, have developed a new class of Ethernet switches based on fabric technology specifically designed for use within today’s dynamic virtualization-driven data centers. These switches are equipped with new features that make better use of inter-switch links, scale out linearly, automatically configure, ensure Quality of Service (QoS), and dynamically adapt to topology changes.

This paper proposes that these technological advances—most specifically the development of Ethernet fabrics based on Transparent Interconnection of Lots of Links (TRILL) or comparable technologies—allow network architects designing networks of moderate scale, up to 200 or so servers, to forgo the large, expensive core switches required by classic Ethernet architectures, replacing them with a fabric of more cost-effective data center switches. The benefits of an Ethernet fabric in the data center network are compelling, including more effective bandwidth utilization, improved network failure recovery, and lower cost of ownership compared to networks using traditional Ethernet architectures.

The paper compares several data center network designs, each supporting 100 physical servers. Those designs are outlined below:

• A full-mesh network using Brocade® VCS Fabric technology • A network using Cisco 6509E and Cisco 4500X switches and a conventional

Spanning Tree Protocol (STP)• A network using Cisco Nexus 7000 core switches with Cisco Nexus 2000 Fabric

ExtendersAfter comparing both cost and performance benchmarks for the three network designs, it was found that a full-mesh network comprising Brocade switches was 43 percent less expensive to acquire than the Cisco Nexus 7000/Cisco Nexus 2000 solution—and less than half the cost of the traditional Cisco Catalyst 6500/ Cisco Catalyst 4500X configuration. In addition, the Brocade network design provided far greater east-west bandwidth between servers and resiliency.*

Brocade VC Fabric

Cisco Catalyst 6509-E/Cisco Catalyst 4500X

Cisco Nexus 7000/Cisco Nexus 2000

$309,260

$767,874

$569,200

$231,945

$575,906

$426,900

$23,125

$134,553

$120,000

10

37

49

5 (1 with BNA)

9

2

5.25:1

Purchase Price5-Year Support

and Maintenance 5-Year Power 5-Year Total Cost Rack Units Devices to ManageUplink Over-subscription

4:1

4:1

$564,330

$1,478,333

$1,106,100

2


Introduction

First-generation local area networks (LANs) were designed primarily to carry data directly between client PCs and file servers or protocol gateways that, in turn, connected to mainframe or supermini computers (for example, DEC’s VAX). This traffic pattern, with most traffic flowing directly between clients and servers, was the de facto standard through the 1990s, when two-tier client server applications were the norm. As data changed from file requests to SQL transactions and queries, the vast majority of traffic flowed north to south between clients and servers—with little traffic, other than back-ups, flowing among servers.

The flow of traffic in today’s data center is markedly different. Applications have been redesigned to allow access from Web browsers and lightweight clients, extending their reach well beyond the traditional fat-client applications of yesteryear. As a result, more network traffic flows between the Web/application servers, where applications run, and database servers, which hold the data, than between the application servers and users.

Virtualization also drives traffic in the east-to-west, server-to-server direction as users run low-bandwidth connections to virtual PCs on VDI servers, and as server administra-tors migrate virtual servers from host to host.

The conventional data center Ethernet design comprising two core switches at the top, an optional layer of aggregation switches below, and edge switches connected to servers, was mandated by the limitations of the Spanning Tree Protocol (STP). STP prevents loops in Ethernet networks by disabling all but one path between each switch in the network at the root bridge. Figure 1 below illustrates those disabled links in red.

Figure 1—A Conventional Network

FastIron WorkgroupFOUNDRYNETWORKS

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

21

22

23

24

17

18

19

20

Console

Power

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

Link

Ac t ivity


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

21

22

23

24

17

18

19

20

Console

Power

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

Link

Ac t ivity


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

21

22

23

24

17

18

19

20

Console

Power

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

Link

Ac t ivity


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

21

22

23

24

17

18

19

20

Console

Power

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

Link

Ac t ivity


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

21

22

23

24

17

18

19

20

Console

Power

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

F DX100

Link / Ac t

Link

Ac t ivity

BigIron 15000F O U N D R YN E T W O R K S BigIron 15000F O U N D R Y

N E T W O R K S

ServerIron 400FOUNDRYNETWORKS




Core

Aggregation

Edge

Servers

XXX

X

XX X X X

3


As a result of STP disabling all but one path from each downstream bridge to the STP root bridge, at least half the uplink bandwidth in any STP network is consumed by idle backup links until something goes wrong. This behavior also makes it impractical to use more than two core switches, since one switch will still be serving as the STP root bridge. Because additional uplinks to incremental core switches are disabled in this model, adding a third or fourth core switch simply increases cost—without actually increasing scalability or bandwidth.

The limitations of the spanning tree protocol, including its hierarchical nature, force network architects to build inefficient networks, adding redundant links and compo-nents that remain idle and are used only in case of another device’s or link’s failure. The multiple layers of switches also cause each packet traveling from server A to server B to traverse multiple switch hops, adding latency to the overall network.

The Emergence of Ethernet Fabrics

Not satisfied with the functionality of standard Ethernet, networking vendors have developed a new technology designed to optimize Ethernet for the today’s new data center environment: Ethernet Fabric.

Ethernet Fabrics

Ethernet fabrics in the data center address the limitations of the 25-year-old STP. While STP solves some of the loop problems present in Ethernet networks, it creates issues of its own. The most obvious of these shortcomings is the waste of potential bandwidth caused by simply blocking paths to prevent loops. STP also has the annoying tendency to choose non-optimal paths, sending data “over the river and through the woods” because the most direct path is blocked. Even the latest Rapid Spanning Tree Protocol can take several seconds to converge when an active link or device fails.

Ethernet fabrics based on TRILL eliminate STP, allowing all links in the network to be active, and take full advantage of the enormous increases in compute power available in today’s switches. These switches recognize not just the active path to any other switch but all other available paths. Like Layer 3 routers, fabric switches use a routing proto-col to build an internal map of all the other switches in the network. Also, like a router, fabric switches can balance traffic loads across all equal-length paths throughout the network—from point A to point B.

Ethernet fabrics make more efficient use of both the intelligence in each switch and the links connecting the switches providing:

•More east-west bandwidth

•Faster network convergence on topology changes like link or switch failures

•Fully automatic convergence on link or switch additions

• Reduced impact of switch failures through distributed rather than hierarchical topologies

4


VM Mobility

As organizations make the transition to server virtualization, they soon discover that they not only get the expected hardware savings that result from server consolidation, they also realize the possibility of significant flexibility gains based on the ability to dynamically migrate virtual servers from host to host.

But dynamic placement of workloads does present a problem for traditional networks that use port profiles—including traffic QoS and Access Control Lists (ACLs)—to deliver the network performance and security required by each workload. When a hypervisor moves a virtual machine (VM) from one host to another, it will now be connected to ports running the wrong profile for that workload.

Switches that incorporate Brocade VCS Fabric technology detect virtual machine movements and automatically migrate port profiles as VMs move, a feature Brocade calls AMPP (automatic migration of port profiles). While AMPP is hypervisor agnostic, Brocade has integrated AMPP with VMware’s vSphere Management Server to automat-ically create virtual server profiles that define the QoS, security settings, and vLANs for virtual machines.

Converged Networking

A new series of Ethernet extensions collectively called Data Center Bridging (DCB) adds the option of lossless transport to Ethernet networks in the data center. The DCB standards define eight priority levels for Ethernet traffic, with a per-priority pause mechanism stopping packets from entering the network if the network is overly congested, along with the protocols that switches use to negotiate their DCB features. This lossless transport mode is used in large networks to support Fibre Channel over Ethernet (FCoE), which encapsulates Fibre Channel Protocol (FCP), into Ethernet frames converging the SAN and LAN into a single network.

In today’s network, DCB’s lossless transport emulates the traffic light at a freeway entrance—smoothing traffic on the network and improving efficiency not just for FCoE but for all network traffic. Any slight delays on ingress to the network introduce less latency than waiting for Transmission Control Protocol (TCP) timeout and subsequent retransmission required when a packet is dropped in a traditional Ethernet.

5


Our Modest Proposal

Most data centers today use the top-of-rack switching topology, which places fixed- configuration switches at the top of each server rack. These switches serve as the edge and/or aggregation layer(s) in the data center network.

Top-of-rack architectures make even more sense as data centers make the transition from 1Gbps to 10Gbps architectures, since the economics of 10Gbps Ethernet dictate keeping cable lengths relatively short. 10Gbase-T promises to address this cable length limitation, but it hasn’t yet reached widespread adoption.

For short distances (up to 15 meters), 10Gbps network architects have the option of using preconfigured direct-attach cables with SFP+ connectors at a cost of $150-$300 each. For longer distances, fiber optic transceivers can be plugged into the same SFP+ socket using fiber optic cable. But 10Gbps transceivers still can cost on the order of $1,000 each—or more than $2,000 per link. As a result, a 10Gbps top-of-rack switch can pay for itself simply based on the reduced cost of optics.

As top-of-rack switches like Brocade VDX® switches were coming to market, we realized that a fabric comprising top-of-rack switches could function as the entire data center network in moderate-sized environments with fewer than roughly 200 servers—with the fabric replacing the function of the traditional core switches. This design promised to not only significantly reduce expenses when compared with a more traditional design, it also had the potential to deliver greater east-west bandwidth at lower latency than a hierarchical network—especially in case of a device failure.

6


The Requirements

To test our theory, we set out to design a data center network for a fictional enterprise—Acme Inc. Its data center contains 100x86 servers, about half of which are currently vSphere hosts. Acme Inc.’s CIO mandated the following simple set of requirements for the new data center.

•100 physical servers

•200 10Gbps converged connections (all dual-homed)

•100 1Gbps Intelligent Platform Management Interface (IPMI) connections

•An additional 50 100/1000Mbps ports for incremental device management connections

•A simple upgrade path to 150 servers

•Each server is to be dual-homed to different switches, so switch failures and firmware upgrades are non-disruptive

•A minimum of 60Gbps of inter-switch bandwidth, with at least 40Gbps remaining after any single device failure.

•Initial configurations must provide 40Gbps of bandwidth from the data center network to the campus network. This must be upgradeable to 80Gbps as needed.

•If they are no longer going to the core of the overall network, Acme Inc. will repurpose two Cisco Catalyst 6509-E switches that currently serve as the core of the network to be the core of the campus network.

•Higher-level services—from inter-vLAN routing to load balancing, firewalls, and other security services—will be provided by the campus network as Acme Inc. expects the majority of east-west traffic to travel on the same vLAN.

•Acme Inc.’s servers are connected to a small number of vLANs/subnets:

—User connections to the majority of servers

—DMZ servers

— Secure servers (holding sensitive data)

—Hypervisor and other management traffic

— vMotion/Live Migration traffic

— iSCSI storage traffic

— FCoE storage traffic

— Backup traffic

The above specifications will be used to test our theory that a set of top-of-rack fabric switches can provide a reliable, high-performance data center network—without the expense of traditional modular core switching gear. This will allow us to compare and contrast this new network design with the following alternatives:

•A 10Gbps network using traditional STP architecture

•A network using Cisco Nexus 7000 Series core switches and Cisco Nexus 2000 Series Fabric Extenders

7


For each network design, we will examine:

•Available east-west bandwidth

•How the network can be expected to behave in case of a device failure

•How complex the initial system configuration will be

•Ongoing cost of vendor maintenance and support

•Data center rack space consumption

•Acquisition cost*

•Power consumption

•The expansion path for each solution to 150 and 300 servers

A network design using a pair of Cisco Nexus 5500 switches with Cisco Nexus 2000 fabric extenders was considered, but as each Nexus 5500 switch would have to carry half the total network traffic. Since the Nexus 5500 doesn’t support high-availability operations like online firmware upgrades or redundant supervisor modules, Acme’s management didn’t feel it offered a high enough level of resiliency.

Juniper’s QFabric was originally considered for this comparison until it was determined that QFabric would cost significantly more than the other options. Creating a QFabric with no single point of failure would require a pair of QFabric Interconnects, the modular switching engine, and a pair of QFabric Directors acting as control servers. With a price tag of more than $300,000 for these central management devices, this network design could be competitive only if amortized over more network ports than were required for this test case.

Brocade VCS Fabric Technology

Our proposed network design connects five 60-port Brocade VDX Switches in a full-mesh fabric using two 10Gbps links between each switch. By enabling Brocade VCS fabric technology to create a Layer 2 multipath fabric from the switches, traffic is load balanced across all equal-length paths through the network.

The “core” of the network is comprised of three Brocade VDX 6720 60-port 10Gbps Ethernet Switches and two Brocade VDX 6730 Switches ,which also have Fibre Channel ports. The network will have two vLANs dedicated to FCoE traffic, with each Brocade VDX 6730 being responsible for connecting one FCoE vLAN to the Fibre Channel switch for Acme Inc’s SAN “A” and the other connecting the second vLAN to SAN “B.”

* Cost analyses are based on manufacturer’s suggested retail prices (MSRP) as published by re-spective vendors. (While we are fully aware that enterprise customers may receive significant discounts from their suppliers, these discounts are not predictable and are usually comparable from vendor to vendor. Volume discounts and customer retention discounts may be available from an incumbent vendor, but prospective vendors may offer similar discounts to get new business. Therefore, MSRP will be used in all comparisons to level the playing field.)

8


Figure 2—The Fabric as Core

Additional Brocade VDX 6710 Switches will support lower-speed devices as well as Intelligent Platform Management Interface (IPMI) management ports for the servers. Since these switches will participate in the Layer 2 multipath fabric, they can be connected in a full- or partial-mesh configuration and can also serve lower-value, or older, servers that don’t require more than 1Gbps per connection.

The Ethernet fabric is attached to Acme Inc.’s campus core with two 10Gbps connections from each of two Brocade VDX 6720 Switches. These are combined to form a single Link Aggregation Control Protocol (LACP) trunk to the core using the Brocade vLAG feature, which allows LACP trunking with connections to separate switches in the fabric.

Cost Analysis

MSRP for the Ethernet fabric solution is $309,260, or $3,092 per server, as of October 2012. Pricing includes twinax cables for Inter-Switch Links (ISLs) and optics for the 40Gbps connection to the campus network.

Since the interconnections between the Brocade VDX top-of-rack switches run between adjacent server racks, we can assume that the vast majority of these connections are less than 15 meters long and can therefore be made with low-cost twinax direct attach cables. By contrast, most core switches are across the data center from servers, necessitat-ing much more expensive fiber optic ISLs.

Each Brocade VDX 6720 and Brocade 6730 Switch will consume two rack units in the server racks where they’re installed. Each Brocade VDX 6710 1Gbps Switch will take 1U, for a total rack space requirement of 13U for the solution.

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6720-60

Brocade VDX 6720

RLOM

51 62 73 84 139 1410 1511 1612 2117 2218 2319 2420

Brocade VDX 6710

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6720-60

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6720-60

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6720-60

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6720-60

Brocade VDX 6720

RLOM

51 62 73 84 139 1410 1511 1612 2117 2218 2319 2420

Brocade VDX 6710Brocade VDX 6720

RLOM

51 62 73 84 139 1410 1511 1612 2117 2218 2319 2420

Brocade VDX 6710

Brocade VDX 6720

RLOM

51 62 73 84 139 1410 1511 1612 2117 2218 2319 2420

Brocade VDX 6710

Brocade VCS Fabric

Purchase Price

5-Year Supportand Maintenance

5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage

Uplink Oversubscription

5 (1 with BNA)

$309,260

$231,945

$23,125

$564,330

10

5.25:1

Cisco Catalyst 6509-ECisco Catalyst 4500X

9

$767,874

$575,906

$134,553

$1,478,333

37

4:1

Cisco Nexus 7000Cisco Nexus 2000

2

$569,200

$426,900

$120,000

$1,106,100

49

4:1

5-Year TCO

$1,478,333

5-Year TCO

$1,106,100

5-Year TCO

$564,330

Purchase Price


5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage


Purchase Price


5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage


9


Assuming an industry standard support and maintenance cost of 15 percent of MSRP, annual maintenance for the solution will be $46,389.

Brocade VCS Fabric technology will use a maximum of 2200 watts of power. Assuming a power usage effectiveness (PUE) of 2 and a power cost of $0.12 per Kwh, the system will cost $4,625 per year for power and cooling.

Based on the above assumptions, the five-year total cost for this network design is $564,330.

Bandwidth and Failure Analysis

Traffic from one server to another across a common switch is forwarded at full wire speed across the Brocade VDX switch’s non-blocking architecture. Traffic between servers that don’t share an active connection to a common switch has access to 80Gbps of aggregate bandwidth—20Gbps of bandwidth from each switch to every other switch. As the switches are connected in a full mesh, the longest data path through the fabric is just two switch hops resulting in total latency of 1.2µs.

With 40 server connections and eight fabric connections per switch, the solution is 5:1 oversubscribed. When all available ports are used for servers, the oversubscription rate rises to 6.5:1 (52 servers per switch).

Increasing fabric bandwidth is simply a matter of adding more ISLs, since Brocade VCS Fabric technology will automatically recognize additional links and begin load balancing traffic across them seconds after they’re installed. Adding a third link to each connection in the Ethernet fabric would boost bandwidth to 120Gbps at a cost of around $1,500. Five 60-port switches with 40Gbps interconnects would make 240 10Gbps Ethernet ports available for use, increase fabric bandwidth to 160Gbps, and increase cost by only about 1 percent more than the starting configuration.

In case of a switch or link failure, the network will reroute traffic over the surviving portion of the network. Since every switch knows the topology of the entire network, the network reconverges many times faster than a STP network, which must rediscover paths whenever a path fails. This reconvergence occurs in a small fraction of a second, allowing standard TCP or SCSI retries to recover with minimal performance impact.

A switch failure in a full-mesh network reduces the available fabric bandwidth by 1/(n-1) where n equals the number of switches in the fabric. For our five-switch fabric, bandwidth will be reduced by 25 percent, to 60Gps, a significantly smaller impact than the 50 percent bandwidth loss if a traditional core switch failed.

The impact on a larger, nine-switch fabric would be only 12.5 percent of the available bandwidth of 160Gbps.

Management and Expansion

Brocade VCS Fabric technology is self-configuring, automatically rebalancing traffic across all available switches and inter-switch links. Network engineers don’t have to spend time designing and configuring multiple Link Aggregation Control Protocol (LACP) connections between the fabric switches, though the campus core to fabric

10


LACP connection via v-LAG will require attention. Once the switches are interconnect-ed and placed in Brocade VCS Fabric technology mode through a single Command-Line Interface (CLI) command, each switch will be automatically assigned a Routing Bridge (Rbridge) number, and the fabric will establish itself automatically.

This solution has a total of eight devices to manage, but the entire fabric can be managed as a single logical switch through Brocade Network Advisor management software. When the software makes a Simple Network Management Protocol (SNMP) connection to a Brocade VCS switch, it recognizes the fabric and automatically adds the rest of the switches to the management.

Figure 3—A VCS Fabric in Brocade Network Advisor

Just as modular switches will identify ports by slot and port, Brocade VCS switch ports are identified, and can be managed, by Rbridge number/slot (which will be 0 for fixed-function switches like those we’re deploying) and port number—allowing an administrator to concurrently manage multiple ports on multiple switches in the fabric.

To grow the network so that it can support the 150 servers that Acme Inc. has specified, we must add an additional Brocade VDX 6720 Switch to the network at an October 2012 list cost of $41,500, including the 10 twinax cables that will be needed to connect the incremental switch to the other switches in the fabric. A six-switch mesh will have 336 usable 10Gbps Ethernet ports.

11


Applicability of the Model Design for Smaller and Larger Configurations

The Ethernet fabric model scales down very well, allowing users with smaller networks to choose between a pair of 60-port switches or a fabric comprising three or four 24-port switches. In this reduced configuration, the failure of a single switch has a smaller impact on network bandwidth.

As the number of switches grows, the number of interconnections in a full-mesh network grows according to the formula n*(n-1) where n equals the number of switches in the mesh. Using 60-port switches and 20Gbps interconnections in a full mesh, the interconnect overhead reaches 30 percent for a 10-switch network—ultimately providing 420 useable ports out of 600.

For networks that need to support 200 or more servers and 400+ connections, the full-mesh architecture is less attractive. For these environments, partial-mesh solutions are more appropriate, most specifically a spine and leaf architecture with several switches making up a central spine and servers connected to leaf switches around the periphery.

A group of four 60-port switches in a 60Gbps full mesh could support an outer ring of 14 switches—with each switch connected to each of the core switches at 20Gbps to provide 936 server access ports.

For 300 servers, we would deploy three Brocade VDX 6720 Switches as the spine, with 14 edge switches to provide 672 server- and storage-facing ports.

Conventional Spanning Tree Solution

Since the Acme Inc. network team didn’t have experience with new data center net-working technologies, its first thought was to build a 10Gbps server network using conventional switches. This Cisco-based solution used the 32-port model from the new Cisco 4500X switch line, with the optional 8-port expansion module. Each Cisco 4500X would use four ports as an LACP trunk (which Cisco calls EtherChannel) to each Cisco Catalyst 6509-E core switch—leaving 32 ports per switch for server connections.

This configuration couldn’t meet the minimum bandwidth requirements using industry-standard switches and STP, since each edge switch would then have only 40Gbps of uplink bandwidth. Cisco does offer two configuration options that would allow all eight uplinks to be active:

•The Cisco virtual switching system (VSS), which allows a pair of Cisco Catalyst 6500 Series Switches to function as one spanning tree bridge

or

•Virtual port channel, which allows LACP links to be homed across multiple servers

Either option will require a few hours of a skilled network engineer’s time to configure.

12


Figure 4—Cisco 6509-E and Cisco 4500-X Solution

Providing the specified 216 10Gbps ports will require seven Cisco Catalyst 4500-X Series Switches, which will, in turn, require 28 ports on each Cisco Catalyst 6509 Switch to be delivered through two 16-port 10Gbps Ethernet cards in each switch. The required 1Gbps ports are provided by a pair of 48-port Gigabit Ethernet cards.

Since Cisco 10Gbps line cards for the Cisco Catalyst 6500 Series Switches use the older XENPAK-style optics, direct-attach twinax cables cannot be used to connect the Cisco Catalyst 4500-X top-of-rack switches to the core. In this deployment scenario, fiber-optic connections must be used—adding significantly to the overall cost.

Cost Analysis

The total acquisition cost for this solution is $767,874 or $7,678 per server—including a new Cisco 6509-E chassis. Assuming an industry standard support and maintenance cost of 15 percent of MSRP, annual maintenance for the solution will total $115,181.

Each Cisco 6509-E chassis occupies 15U of rack space, while the Cisco Catalyst 4500-X Switches occupy 1U each. Total rack space consumed is therefore 37U.

Each Cisco 6509-E chassis will draw somewhere around 5000W while the Cisco Catalyst 4500-X switches draw up to 400W each for a total power consumption of 12,800W. Assuming a PUE of 2 and a power cost of $0.12 per Kwh, the system will cost $26,910 per year for power and cooling.

Brocade VCS Fabric

Purchase Price


5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage


5 (1 with BNA)

$309,260

$231,945

$23,125

$564,330

10

5.25:1


9

$767,874

$575,906

$134,553

$1,478,333

37

4:1


2

$569,200

$426,900

$120,000

$1,106,100

49

4:1

5-Year TCO

$1,478,333

5-Year TCO

$1,106,100

5-Year TCO

$564,330

Purchase Price


5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage


Purchase Price


5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage


13


Based on the above assumptions, the five-year total cost for this network design is $1,478,333.

If Acme Inc. chooses to upgrade the supervisors and install line cards in its existing Cisco Catalyst 6500 chassis to continue to have a single network core, it could save approximately $40,000. This would make sense only if its current switches had sufficient slots available for line cards.


Traffic among servers connected to the same Cisco Catalyst 4500-X Series Switch can run at wire rate, while all other traffic will take three switch hops as it passes through the source, core, and destination switches over the 40Gbps trunk to each core switch.

These links are 4:1 oversubscribed under normal conditions (32 servers:8 uplinks), creating a potential bottleneck. Even worse, the Cisco WS-X6816 line card is itself 4:1 oversubscribed, so the end-to-end path could be as much as 8:1 oversubscribed.

An alternative design would use an 8-port Cisco WS-X6908 line card, which is not over-subscribed to the switch fabric, but that would necessitate using a Cisco Catalyst 6513-E Switch Chassis to have any slots free for future growth—raising the cost of the solution to almost $1 million.

In case of a core switch failure, the oversubscription rate for the whole network will double to 8:1. In case of an edge switch failure, the server’s Network Interface Controller (NIC) teaming or LACP connection will failover to the server’s connection to another switch.


This solution has nine devices to manage and approximately 60 inter-switch connections that must be manually configured into LACP trunks. Initial configuration of the network will require at least a full day’s work by a skilled engineer.

Expanding the network to 150 servers will require two additional Cisco Catalyst 4500-X Series edge switches and two additional 10Gbps line cards for the core Cisco Catalyst 6509 Switches. This expansion will cost $201,964.

Expanding to 300 servers would add five Cisco Catalyst 4500-X Series Switches and another pair of 10Gbps line cards.

Cisco Nexus 7000 with Cisco Nexus 2000 Fabric Extenders

The Nexus series represents Cisco’s entry into the market for next-generation data center switches. The first Nexus design we will evaluate comprises a pair of Cisco Nexus 7000 modular switches and the Cisco Nexus 2000 Fabric Extender. These solutions work together to provide connectivity to servers and other devices on the data center network.

Each Cisco Nexus 2232 has 32 useable ports and eight 10Gbps Ethernet ports for upstream connections to one or two Cisco Nexus 7000 or Cisco Nexus 5000 switches. We used seven Cisco Nexus 2232s to provide 224 10Gbps Ethernet ports for server and storage connections.

14


48-port 10/100/1000 line cards in the Cisco Nexus 7000 were used to provide a total of 192 twisted-pair Ethernet ports. While Acme Inc.’s design requirement called for only 140 lower-speed Ethernet ports, which would require only three cards, in our experi-ence, most network designers would add the matching cards to both of the core switches rather than use two cards in one switch and one in the other.

N2K- 2232TM

STAT

ID

25 26 27 28 29 30 31 3217 18 19 20 21 22 23 249 10 11 12 13 14 15 169 2 3 4 5 6 7 81 2 3 4 5 6 7 8

N2K- M2800P

N2K- 2232TM

STAT

ID

25 26 27 28 29 30 31 3217 18 19 20 21 22 23 249 10 11 12 13 14 15 169 2 3 4 5 6 7 81 2 3 4 5 6 7 8

N2K- M2800P

N2K- 2232TM

STAT

ID

25 26 27 28 29 30 31 3217 18 19 20 21 22 23 249 10 11 12 13 14 15 169 2 3 4 5 6 7 81 2 3 4 5 6 7 8

N2K- M2800P

N2K- 2232TM

STAT

ID

25 26 27 28 29 30 31 3217 18 19 20 21 22 23 249 10 11 12 13 14 15 169 2 3 4 5 6 7 81 2 3 4 5 6 7 8

N2K- M2800P

N2K- 2232TM

STAT

ID

25 26 27 28 29 30 31 3217 18 19 20 21 22 23 249 10 11 12 13 14 15 169 2 3 4 5 6 7 81 2 3 4 5 6 7 8

N2K- M2800P

N2K- 2232TM

STAT

ID

25 26 27 28 29 30 31 3217 18 19 20 21 22 23 249 10 11 12 13 14 15 169 2 3 4 5 6 7 81 2 3 4 5 6 7 8

N2K- M2800P

N2K- 2232TM

STAT

ID

25 26 27 28 29 30 31 3217 18 19 20 21 22 23 249 10 11 12 13 14 15 169 2 3 4 5 6 7 81 2 3 4 5 6 7 8

N2K- M2800P

PSU FAN SUP FAB IOM

Cisco Nexus 7000 Series

1 2 3 4 5 6 7 8 9 10

STATUSID

12

34

56

78

910

1112

1314

1516

1718

1920

2122

2324

2526

2728

2930

3132

3334

3536

3738

3940

4142

4344

4546

4748

PORT GROUP 1PORT GROUP 12

PORT GROUP 11





PORT GROUP 10N7K-F248XP-25

12

1122

1314

2324

2526

3536

3738

4748

1213

1415

1617

1819

2021

2223

2425

2627

2829

3031

3233

3435

3637

3839

4041

4243

4445

4647

48

N7K

-M148G

T-111

211

1213

1423

2425

2635

3637

3847

48

STA

TUS

1213

1415

1617

1819

2021

2223

2425

2627

2829

3031

3233

3435

3637

3839

4041

4243

4445

4647

48

N7K

-M148G

T-111

211

1213

1423

2425

2635

3637

3847

48

STA

TUS

STATUS

ID

N7K-SUP1

SYSTEMACTIVEPWR MGMT

ACT

LINK

SERIA

LPO

RT

CO

M1/A

UX

SERIA

LPO

RT

CO

NSO

LE

ACT

LINK

12

HO

STPO

RTS

USB

DEVIC

EPO

RT

CMPSTATUS

CMPMGMT

ETH

RESET

STATUS

ID

N7K-SUP1


ACT

LINK

SERIA

LPO

RT

CO

M1/A

UX

SERIA

LPO

RT

CO

NSO

LE

ACT

LINK

12

HO

STPO

RTS

USB

DEVIC

EPO

RT

CMPSTATUS

CMPMGMT

ETH

RESET

Nexus 7010

PSU FAN SUP FAB IOM

Cisco Nexus 7000 Series

1 2 3 4 5 6 7 8 9 10

STATUSID

12

34

56

78

910

1112

1314

1516

1718

1920

2122

2324

2526

2728

2930

3132

3334

3536

3738

3940

4142

4344

4546

4748


PORT GROUP 11





PORT GROUP 10N7K-F248XP-25

12

1122

1314

2324

2526

3536

3738

4748

1213

1415

1617

1819

2021

2223

2425

2627

2829

3031

3233

3435

3637

3839

4041

4243

4445

4647

48

N7K

-M148G

T-111

211

1213

1423

2425

2635

3637

3847

48

STA

TUS

1213

1415

1617

1819

2021

2223

2425

2627

2829

3031

3233

3435

3637

3839

4041

4243

4445

4647

48

N7K

-M148G

T-111

211

1213

1423

2425

2635

3637

3847

48

STA

TUS

STATUS

ID

N7K-SUP1


ACT

LINK

SERIA

LPO

RT

CO

M1/A

UX

SERIA

LPO

RT

CO

NSO

LE

ACT

LINK

12

HO

STPO

RTS

USB

DEVIC

EPO

RT

CMPSTATUS

CMPMGMT

ETH

RESET

STATUS

ID

N7K-SUP1


ACT

LINK

SERIA

LPO

RT

CO

M1/A

UX

SERIA

LPO

RT

CO

NSO

LE

ACT

LINK

12

HO

STPO

RTS

USB

DEVIC

EPO

RT

CMPSTATUS

CMPMGMT

ETH

RESET

Nexus 7010

Nexus 2000 Fabric Extenders

Figure 5—Cisco Nexus 7010 and Cisco Nexus 2232 Network

Even though it’s not required, with just two Cisco Nexus 7000 switches, we are configuring the core switches to use Cisco FabricPath (a Cisco pre-standard TRILL implementation that automatically configures the inter-switch links and load balances across them).

Cost Analysis

MSRP for this solution is $569,200 or $5,692 per server.

The Cisco Nexus 7010 switches are each 21 rack units tall, while the Cisco Nexus 2232s are 1U each. The total solution will require 49 rack units, including a full 42U rack for each Cisco Nexus 7010 alone.

Assuming the industry standard 15 percent an-nual cost for support and maintenance, a support contract for this solution will cost $85,380 a year or $426,900 over the five-year projected life of the network.

Brocade VCS Fabric

Purchase Price


5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage


5 (1 with BNA)

$309,260

$231,945

$23,125

$564,330

10

5.25:1


9

$767,874

$575,906

$134,553

$1,478,333

37

4:1


2

$569,200

$426,900

$120,000

$1,106,100

49

4:1

5-Year TCO

$1,478,333

5-Year TCO

$1,106,100

5-Year TCO

$564,330

Purchase Price


5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage


Purchase Price


5-Year Power

5-Year Total Cost

Rack Units

Devices to Manage


15


Each Cisco Nexus 7010 will consume around 5000W of power and each Cisco Nexus 2232 about 225W. The total power use for the system is 11575W. Assuming a PUE of 2 and power costs of $0.12 per Kwh, the system will cost $24,000 for power and cooling.

Based on the above assumptions, the five-year total cost for this network design is $1,106,100.

An alternative configuration would use six Cisco Nexus 2224 fabric extenders to provide low-speed connections. This approach would reduce the cost of the two Cisco Nexus-based configurations by $12,000.

While it’s likely that the Cisco Nexus 2232s at the top of each server rack will be across the data center from the Cisco Nexus 7010s and require fiber-optic connections, we’ve priced this configuration using twinax cables for all inter-switch links to avoid inflating the cost for those data centers where the longest links are less than 15 meters.

About the Cisco Nexus 2000 Fabric Extender

While the Cisco Nexus 2232 Fabric Extender is a 1U box with SFP+ 10Gbps Ethernet ports, it is not an Ethernet switch. Fabric extenders function more like remote line cards than true switches; all traffic from the 32 downlink ports on the Cisco Nexus 2232 is sent upstream to the Cisco Nexus 7000 or Cisco Nexus 5000 parent switch for processing—even if the source and destination are ports on the same fabric extender.

As a result, while it may appear that a solution based on the Cisco Nexus 2232 with 32 server-facing ports and eight uplinks, and a solution based on the Cisco Catalyst 4500-X, which also has 32 user ports and eight ports for inter-switch links, are both 4:1 oversubscribed—the switch actually offers significantly more bandwidth.

If two servers connected to the same switch want to exchange data, the switch forwards packets between ports, consuming none of the bandwidth on its inter-switch link ports. If those two servers were connected to a Cisco Nexus 2000, traffic would be routed first to the parent switch and then back to the Cisco Nexus 2000. In fact, traffic between ports on the same Cisco Nexus 2000 uses twice the ISL bandwidth as traffic going to a port on anoth-er Cisco Nexus 2000 since it travels to the parent switch and back again.

Since we can’t quantify the amount of traffic that exists between ports on the same switch, and because the advantages of workload mobility in today’s data center make keeping related workloads on the same switch counter-productive—we use a worst-case scenario when describing a switch such as Cisco Catalyst 4500-X or a Brocade VDX 6720, as 4:1 oversubscribed, which makes this comparison somewhat misleading.

16


An alternative configuration would use six Cisco Nexus 2224 fabric extenders to provide low-speed connections. This approach would reduce the cost of the two Cisco Nexus-based configurations by $12,000.


Each Cisco Nexus 2232 has 32 ports for server connections and eight uplinks to the Cisco Nexus 7010s, making each link oversubscribed 4:1. Since all traffic to and from servers will pass over the Cisco Nexus 2232 uplinks, these connections are likely to become congested.

In case of a core switch failure, half of the network’s east-west bandwidth is lost, bringing the oversubscription rate to 8:1. It’s common practice to connect the first Ethernet port on servers in racks “A” and “B” to the switch in rack “A”, and the second to the switch in rack “B”. As a result, a failure of a fabric extender would cause all server traffic in both racks to be shifted to the same fabric extender—again raising the oversubscription rate to 8:1.


Since fabric extenders are seen and managed as if they were remote line cards for their parent switches, this configuration really has only two points of management—the Cisco Nexus 7010 Switches. Since we’ve chosen to use Cisco FabricPath, the inter-switch TRILL will detect the inter-switch links and configure them.

Growing to 150 servers will require adding two more Cisco Nexus 2232 Fabric Extend-ers. Since Cisco FabricPath creates fabrics of Cisco Nexus 7000 Switches (each with Cisco Nexus 2000s attached), this architecture can be expanded to thousands of 10Gbps Ethernet ports.

This alternative approach would reduce the cost of the two Cisco Nexus-based configurations by $12,000.

As in the first Cisco Nexus configuration we’ve priced, this configuration uses twinax cables for all inter-switch links to avoid inflating the cost for those data centers where the longest links are less than 15 meters.

17


Lab Testing

Our testing concentrated on the behavior of an application running across a Brocade VCS fabric technology-based switch cluster during a switch failure. We ran the Dell DVD Store application to provide a realistic application load—simulating users connect-ing to an Internet store selling DVDs. Like many Web applications, the DVD Store uses an application server to handle the business processes and user interface, combined with a database server.

For our testing, we used virtual application and database servers on separate physical servers running VMware vSphere 4.1. The physical servers were configured to use the network interface on each server that was connected to a common switch as the primary interface, with failover to secondary interfaces on different switches.

We started up the DVD Store application and then cut the power to the common switch that was carrying the data to see how quickly the VCS fabric and VMware vSphere NIC teaming would recover—and how this failure would impact the application.

Figure 6—DVD Store Performance

Next, we created a modified version of the DVD Store application to record the number of operations per second, the maximum round-trip transaction latency, and the aver-age round-trip transaction latency once per second (rather than the default every 10 seconds). Note that what DVD Store is reporting here is transaction latency—including SQL server access, not just network latency.

18


As you can see in Figure 6 above, when the switch that was in the data path was powered down, the maximum latency jumped to 921ms, which indicated that some frames were lost as the switch failed, but the system failed over to the alternate path so quickly that while latency peaked at 921ms, average latency for that second remained 9ms, indicating a very small number of lost frames. More important, the rate at which the application processed DVD sales was essentially unchanged at 5,827 operations per second compared to an average of 5,832 per second for the 10-minute test run.

Given the low impact of the switch failure on application performance, we set up Solarwinds Real-Time Bandwidth Monitor to make sure data were actually taking the paths we had configured them to take. When we graphed the traffic levels for the four ports connected to our test servers and ran our test again, we saw the screen shown below as Figure 7.

Figure 7—SNMP Traces of Network Traffic During Test

In this screenshot, the top two graphs illustrate the traffic running through the common switch, and the bottom two graphs show the ports on the switch for the backup connec-tions on the servers. As the primary switch goes offline, you can see the traffic rerouted to the alternate path. Since these graphs are generated from SNMP data collected in each switch, there are no data for the period when the default switch is powered down. Also, note when it comes back online, the traffic is rerouted back to its default path.

19


When analyzing the data, we calculated that for a peak latency of 920ms to have no effect on the average latency of 9ms, there must have been only a very small number of packets lost and retransmitted. To figure out exactly how many, we used Windows Performance Monitor to log TCP retransmits per second and TCP segments per second on both virtual servers. On each system, the peak number of retransmissions was four, reinforcing our view that very few frames were dropped in the failover.

Figure 8—SQL Server TCP Counters

20


Our Test Configuration

We connected three Brocade VDX 6720 Switches and two Brocade VDX 6730 Switches to create a fully meshed Brocade VCS cluster with a single 10Gbps link from each switch to all other switches. We then connected a pair of SuperMicro 5017C-TF servers running VMware vSphere 4.1U2 to the cluster. One 10Gbps Ethernet port on each server was connected to a common switch, which we called Brocade switch “E” and the second port on each of the two servers was connected to a different switch.

12

UID

RESET

Application ServerHost

12

UID

RESET

SQL Server Host

E

D A

BC

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6720-60

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6720-60

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6720-60

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6730-60

Brocade VDX 6720

RLOM

61 72 83 94 1611 1712 1813 1914 2621 2722 2823 2924 3631 3732 3833 3934105 2015 3025 4035

4641 4742 4843 4944 5651 5752 5853 59545045 6055

Brocade VDX 6730-60

Figure 9—Lab Test Configuration

Both ports of the 10Gbps card in each host were assigned to a dedicated vSwitch in that host. We configured VMware vSphere NIC teaming to use the connection to Brocade switch “E” as the primary connection, and use the other connection for failover.

vSphere Host Server Hardware

Processor Intel Xeon E3-1230

(3.2Ghz Quad-Core)Memory 16Gb DDR3-133310Gbps NIC Emulex OCe10102 (SQL Server)

Intel X520 (Application Server)

A virtual Windows 2008 R2 server running Microsoft SQL Server 2008 was placed on one host as the only active guest. A similar virtual server running the Dell DVD Store application server was the only active guest on the second host server.

21


Virtual DVD Store Server Configuration

Processor 4 vCPU

Memory 10GbNetwork Interface 1 vNIC to management vSwitch

1 vNIC to vSwitch connected to VCS mesh

We modified the DVD Store application to report statistics once per second, rather than the default of once every 10 seconds. We ran the application for five minutes to establish baseline performance, powered down the Brocade switch “E” and then powered it back on.

We used the DVD Store application’s logging SNMP data from the switches and Windows Performance Monitor to examine the system’s behavior throughout the process.

About the DVD Store Application

Dell DVD Store application simulates an online store selling DVDs. An application server makes requests simulating users connecting to the store’s website and accesses the database server to retrieve titles and record sales. The DVD Store application has been widely used as a benchmarking tool and is included as part of the VMware VMark 2 benchmark suite.

A Dell white paper describing the DVD Store application in detail can be downloaded at http://www.dell.com/downloads/global/power/ps3q05-20050217-Jaffe-OE.pdf and an overview video is available on YouTube at http://www.youtube.com/watch?v=4tHT-5TOrDQ. The application can be downloaded at http://linux.dell.com/dvdstore/.

22


Conclusions

The development of Ethernet fabric technologies has freed data center network architects from the limitations imposed by the Spanning Tree Protocol. By using a fabric of top-of-rack switches, network architects can build a network that provides all the performance and reliability of a conventional edge, aggregation, and core network at a much lower cost.

We compared the cost, performance, and complexity of several data center network designs to support 100 servers—each with two 10Gbps Ethernet connections. The Brocade VCS fabric solution was generally half the cost of the more conventional networks, based on either Cisco’s traditional Catalyst or data center-oriented Nexus switches, while providing a comparable amount of server-to-server bandwidth. The Brocade solution also allows users to simply and inexpensively add more interswitch bandwidth, at any time, by simply adding more interconnects, while the other solutions were at their respective limits.

Summary of the Three Network Configurations

Even the least expensive Cisco configuration, using Cisco Nexus 7000 switches with Cisco Nexus 2000 Fabric extenders, was almost twice the cost of the Brocade VCS solu-tion. Readers should also note that this cost analysis does not include soft costs such as staff or data center real estate. These costs are too user specific to be easily quantifiable. Also, while the ease of configuration and smaller size of the fabric network may save users additional money, we haven’t included these potential savings in our analysis.

Cost isn’t the only advantage of an Ethernet fabric-as-core solution. The Ethernet fabric approach lets network architects create small and large networks from the same build-ing blocks. Using a spine and leaf architecture, an Ethernet fabric network of 60-port Brocade VDX 6730 switches could support several hundred ports, with a maximum of three switch hops from point to point—rather than the five hops of a traditional edge-aggregation-core design.

The Ethernet fabric network is also more resilient than a hierarchical network. Total network bandwidth in a traditional network, or one based on fabric extenders, can be cut in half when a critical device like a core switch goes offline. Since all the switches in the fabric are peers, the loss of one switch in a five-switch network causes only a 20 percent loss of capacity.

Brocade VC Fabric

Cisco Catalyst 6509-E/Cisco Catalyst 4500X

Cisco Nexus 7000/Cisco Nexus 2000

$309,260

$767,874

$569,200

$231,945

$575,906

$426,900

$23,125

$134,553

$120,000

10

37

49

5 (1 with BNA)

9

2

5.25:1

Purchase Price5-Year Support

and Maintenance 5-Year Power 5-Year Total Cost Rack Units Devices to ManageUplink Over-subscription

4:1

4:1

$564,330

$1,478,333

$1,106,100

23


Since it doesn’t rely on Spanning Tree Protocol, the Ethernet fabric network also converges faster than a traditional network. Even with the latest Rapid Spanning Tree implementation, it can take up to six seconds for a network to recover from a switch or link loss. Our testing showed that the Brocade VCS fabric network recovers in a matter of milliseconds. Faster recovery means less lost data for higher-layer protocols like TCP or SCSI to recover. In our testing, only one frame was lost when the switch serving the active connection between our servers failed.

In short, an Ethernet fabric network using only top-of-rack switches can perform better than a traditional network using modular switches in the core. This architecture is:

•Approximately half the cost to:

— Purchase

— Power

— Maintain

•More flexible

— Cross-network bandwidth can be added by increased connections

— The same building blocks can be used for networks small and large

•More resilient

— Less bandwidth loss on failure

— Faster convergence on failure

24


Appendix A – Network Configuration Bills of Materials

Brocade VCS Configuration

Product Description Qty Unit Price

Dis-count

%Final Price Total Price

BR-VDX6710-54-F

Brocade VDX 6710, 48P GbE, 6P SFP+, AC, non-port side exhaust airflow 3 9,500.00 0.00 9,500.00 28,500.00

BR-VDX6710-54VCS-01 VCS software license for Brocade VDX 6710-54 3 3,600.00 0.00 3,600.00 10,800.00

BR-VDX6720-60-F Brocade VDX 6720, 60P SFP+, AC, non-port side exhaust airflow 3 40,200.00 0.00 40,200.00 120,600.00

BR-VDX6720-60VCS-01 VCS software license for Brocade VDX 6720-40, Brocade VDX 6720-60 3 9,000.00 0.00 9,000.00 27,000.00

BR-VDX 6720-60FCOE-01 FCoE software license for 40- and 60-port SKU 3 6,960.00 0.00 6,960.00 20,880.00

BR-VDX6730-60-FBrocade VDX 6730, 60P SFP+, AC, non-port side exhaust airflow 2 40,200.00 0.00 40,200.00 80,400.00

BR-VDX6730-60VCS-01 VCS software license for Brocade VDX 6730-40, Brocade VDX 6730-60 2 6,960.00 0.00 6,960.00 13,920.00

10G-SFPP-SR (1-pack) 10 Gbps SR Optics 4 1,040.00 0.00 1,040.00 4,160.00

10G-SFPP-TWX-0301

(1-pack)10 Gbps SFP+ direct-attached cables, 3 m Twinax copper 20 150.00 0.00 150.00 3,000.00

Data Products and Services Total 309,260.00

First year Maintenance (at 15%) $46,389.00

First year Power $4,625.28

First year Total $360,274.28

Maintenance years 2-5- $185,556.00

Power years 2-5 $18,501.12

Five year total cost $564,331.40

25


Cisco 6509-E/Cisco 4500-X Configuration


Dis-count

%Final Price Total Price

VS-S2T-10G Supervisor 2T for C6500 2 25,000.00 0.00 25,000.00 50,000.00

VS-6509E-SUP2TCatalyst 6509-E Supervisor 2T bundle 2 38,000.00 0.00 38,000.00 76,000.00

WS-X6816-10G-2T16 Port XENPAK 10Gbps Ethernet line card 4 38,000.00 0.00 38,000.00 152,000.00

8.7KW Power supply for 6500 2 7000 0.00 7,000.00 14,000.00

WS-C4500X-32SFP+Catalyst 4500-X ToR switch 32 ports 7 40,000.00 0.00 40,000.00 280,000.00

C4KX-NM-8SFP+8 port expansion module for C450-X 7 7,000.00 0.00 7,000.00 49,000.00

C4KX-PWR-750AC-F C4500-X redundant power supply 7 2,000.00 0.00 2,000.00 14,000.00

X2-10GB-SR X2 10GBase-SR transceiver 56 3,000.00 65.00 1,050.00 58,800.00

SFP-10G-SR= 10GBase-SR SFP+ 56 1,495.00 55.00 672.75 37,674.00

Fiber-Optic Cables 56 25.00 0.00 25.00 1,400.00

WS-X6748-GE-TX48 port 10/100/1000 Ethernet module 4 10,350.00 0.00 10,350.00 41,400.00

Data Products Total $774,274.00

First year maintenance (at 15%) $116,141.10

First year Power $26,910.72

First Year Total $917,325.82

Maintenance years 2-5 $464,564.40

Power years 2-5 $107,642.88

Five year total costs $1,489,533.10

Notes on Cisco 6509-E/Cisco 4500-X Configuration

This configuration includes optics and fiber cables for connecting the Cisco 6509-E and Cisco 4500-X Switches as twinaxial cables cannot be used with the Cisco 6509-E’s XEN-PACK interfaces. The discounts shown for those components are roughly those that can be found on the Web.

26


Cisco Nexus 7000/ Cisco Nexus 2000


Dis-count

%

Final Price

Total Price

N7K-C7010-BUN2Nexus 7010 Bundle (Chassis,SUP1,(5)FAB2, (2)AC-6KW PSU) 2 79,000.00 0.00 79,000.00 158,000.00

N7KS1K9-60 Cisco NX-OS Release 6.0 2 0.00 0.00 0.00 0.00

N7K-ADV1K9Nexus 7000 Advanced LAN Enterprise License (VDC, CTS ONLY) 2 20,000.00 0.00 20,000.00 40,000.00

N7K-F248XP-25Nexus 7000 F2-Series 48 Port 10GbE (req. SFP+) 2 44,000.00 0.00 44,000.00 88,000.00

N7K-M148GT-11 Nexus 7000 - 48 Port 10/100/1000, RJ-45 4 15,000.00 0.00 15,000.00 60,000.00

N7K-SUP1Nexus 7000 - Supervisor, Includes External 8GB Log Flash 2 25,000.00 0.00 25,000.00 50,000.00

N7K-SUP1-8GBUPGNexus 7000 Supervisor 1 8GB Memory Upgrade Kit 2 Included 0.00 Included Included

N7K-SUP1-BUNNexus 7000 - Supervisor 1, Includes External 8GB Flash 2 Included 0.00 Included Included

N7K-C7010-FAB-2Nexus 7000 - 10 Slot Chassis - 110Gbps/Slot Fabric Module 10 Included 0.00 Included Included

N7K-AC-6.0KWNexus 7000 - 6.0KW AC Power Supply Module 2 6,000.00 0.00 6,000.00 12,000.00

N7K-AC-6.0KWNexus 7000 - 6.0KW AC Power Supply Module 2 Included 0.00 Included Included

CAB-7513AC AC Power Cord North America (110V) 6 0.00 0.00 0.00 0.00

N7K-EL21K9Nexus 7000 Enhanced Layer 2 License (FabricPath) 2 25,000.00 0.00 25,000.00 50,000.00

CAB-9K12A-NAPower Cord, 125VAC 13A NEMA 5-15 Plug, North America 16 0.00 0.00 0.00 0.00

SFP+ Cables 64 150.00 0.00 150.00 9,600.00

N2K-C2232PP-10GEN2K 10GE, 2 AC PS, 1 Fan (Std Air), 32x1/10GE+8x10GE 7 9,000.00 0.00 9,,000.00 63,000.00

Data Products Total 521,000.00

First Year Maintenance 78,150.00

First Year Power 24,335.28

First Year Total 667,875.28

Maintenance years 2-5 312.6000

Power years 2-5 97,341.12

Five year total costs 1,100,976.40

Flattening the Data Center Network - Bitpipe

Documents

Transcript of Flattening the Data Center Network - Bitpipe