Nội dung

Campus Network for High Availability Design Guide Cisco Validated Design I January 25, 2008 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 Customer Order Number: Text Part Number: OL-15734-01 Cisco Validated Design The Cisco Validated Design Program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit www.cisco.com/go/validateddesigns. ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO. CCVP, the Cisco Logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco Systems, Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me Browsing, FormShare, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys, MeetingPlace, MGX, Networking Academy, Network Registrar, Packet, PIX, ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet, StackWise, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0612R) Campus Network for High Availability Design Guide © 2007 Cisco Systems, Inc. All rights reserved. C O N T E N T S Introduction 1-1 Audience 1-1 Document Objectives 1-1 Campus Network Design Overview 1-2 Design Recommendations Summary 1-2 Tuning for Optimized Convergence 1-2 Access Layer Tuning 1-2 Distribution Layer Tuning 1-3 Core Layer Tuning 1-4 Design Guidance Review 1-4 Layer 3 Foundations Services 1-4 Layer 2 Foundation Services 1-5 General Design Considerations 1-6 Hierarchical Network Design Model 1-7 Hierarchical Network Design Overview Core Layer 1-8 Distribution Layer 1-9 Access Layer 1-10 Network and In-the-Box Redundancy 1-7 1-11 Foundation Services Technologies 1-14 Layer 3 Routing Protocols 1-14 Using Triangle Topologies 1-14 Limiting L3 Peering to Transit Links 1-15 Ensuring Connectivity in Case of Failure 1-16 Tuning Load Balancing with Cisco Express Forwarding 1-19 Layer 2 Redundancy—Spanning Tree Protocol Versions 1-22 Spanning Tree Protocol Versions 1-22 Best Practices for Optimal Convergence 1-23 Trunking Protocols 1-25 Deploying Multiple VLANS on a Single Ethernet Link (Trunking) Virtual Trunk Protocol 1-27 Dynamic Trunk Protocol 1-28 Preventing Double 802.1Q Encapsulated VLAN Hopping 1-29 1-26 Campus Network for High Availability Design Guide OL-15734-01 i Contents Protecting Against One-Way Communication with UniDirectional Link Detection Link Aggregation—EtherChannel Protocol and 802.3ad 1-32 Link Aggregation Protocol 1-33 Using HSRP, VRRP, or GLBP for Default Gateway Redundancy 1-35 Gateway Load Balancing Protocol 1-37 Oversubscription and QoS 1-40 1-31 Design Best Practices 1-43 Daisy Chaining Dangers 1-43 Asymmetric Routing and Unicast Flooding 1-45 Designing for Redundancy 1-47 Spanning VLANs Across Access Layers Switches 1-51 Deploying the L2 /L3 Boundary at the Distribution Layer 1-51 Routing in the Access Layer 1-53 Deploying the L2/L3 Boundary at the Access Layer 1-53 Comparing Routing Protocols 1-55 Using EIGRP in the Access Layer 1-56 Using OSPF in the Access Layer 1-57 Summary 1-59 Campus Network for High Availability Design Guide ii EDCS-569061 Campus Network for High Availability Design Guide Introduction This document is the first in a series of two documents describing the best way to design campus networks using the hierarchical model. The second document, High Availability Campus Recovery Analysis, provides extensive test results showing the convergence times for the different topologies described in this document, and is available at the following website: http://www.cisco.com/application/pdf/en/us/guest/netsol/ns432/c649/cdccont_0900aecd801a89fc.pdf This document includes the following sections: • Campus Network Design Overview, page 2 • Design Recommendations Summary, page 2 • Hierarchical Network Design Model, page 7 • Network and In-the-Box Redundancy, page 11 • Foundation Services Technologies, page 14 • Design Best Practices, page 43 • Summary, page 59 Audience This document is intended for customers and enterprise systems engineers who are building or intend to build an enterprise campus network and require design best practice recommendations and configuration examples. Document Objectives This document presents recommended designs for the campus network, and includes descriptions of various topologies, routing protocols, configuration guidelines, and other considerations relevant to the design of highly available and reliable campus networks. Corporate Headquarters: Cisco Systems, Inc., 170 West Tasman Drive, San Jose, CA 95134-1706 USA Copyright © 2007 Cisco Systems, Inc. All rights reserved. Campus Network Design Overview Campus Network Design Overview Designing a campus network may not appear as interesting or exciting as designing an IP telephony network, an IP video network, or even designing a wireless network. However, emerging applications like these are built upon the campus foundation. Much like the construction of a house, if the engineering work is skipped at the foundation level, the house will crack and eventually fail. If the foundation services and reference design in an enterprise network are not rock-solid, applications that depend on the services offered by the network like IP telephony, IP video, and wireless communications will eventually suffer performance and reliability challenges. To continue the analogy, if a reliable foundation is engineered and built, the house will stand for years, growing with the owner through alterations and expansions to provide safe and reliable service throughout its life cycle. The same is true for an enterprise campus network. The design principles and implementation best practices described in this document are tried-and-true lessons learned over time. Your enterprise can take advantage of these lessons to implement a network that will provide the necessary flexibility as the business requirements of your network infrastructure evolve over time. Design Recommendations Summary This section summarizes the design recommendations presented in this document and includes the following topics: • Tuning for Optimized Convergence, page 2 • Design Guidance Review, page 4 Tuning for Optimized Convergence This section summarizes the recommendations for achieving optimum convergence in the access, distribution, and core layers, and includes the following topics: • Access Layer Tuning, page 2 • Distribution Layer Tuning, page 3 • Core Layer Tuning, page 4 Access Layer Tuning The following are the recommendations for optimal access layer convergence: • Limit VLANs to a single closet whenever possible. There are many reasons why STP/RSTP convergence should be avoided for the most deterministic and highly available network topology. In general, when you avoid STP/RSTP, convergence can be predictable, bounded, and reliably tuned. Additionally, it should be noted that in soft failure conditions where keepalives (BPDU or routing protocol hellos) are lost, L2 environments fail open, forwarding traffic with unknown destinations on all ports and causing potential broadcast storms; while L3 environments fail closed, dropping routing neighbor relationships, breaking connectivity, and isolating the soft failed devices. • If STP is required, use Rapid PVST+. Campus Network for High Availability Design Guide 2 OL-15734-01 Design Recommendations Summary If you are compelled by application requirements to depend on STP to resolve convergence events, use Rapid PVST+. Rapid PVST+ is far superior to 802.1d and even PVST+ (802.1d plus Cisco enhancements) from a convergence perspective. • Set trunks to on/on with no negotiate, prune unused VLANs, and use VTP transparent mode. When configuring switch-to-switch interconnections to carry multiple VLANs, set DTP to on/on with no negotiate to avoid DTP protocol negotiation. This tuning can save seconds of outage when restoring a failed link or node. Unused VLANs should be manually pruned from trunked interfaces to avoid broadcast propagation. Finally, VTP transparent mode should be used because the need for a shared common VLAN database is reduced. • Match PAgP settings between CatOS and Cisco IOS software. When connecting a Cisco IOS software device to a CatOS device, make sure that PAgP settings are the same on both sides. The defaults are different. CatOS devices should have PAgP set to off when connecting to an Cisco IOS software device if EtherChannels are not configured. • Consider EIGRP/Routing in the access layer. A routing protocol like EIGRP, when properly tuned, can achieve better convergence results than designs that rely on STP to resolve convergence events. A routing protocol can even achieve better convergence results than the time-tested L2/L3 boundary hierarchical design. However, some additional complexity (uplink IP addressing and subnetting) and loss of flexibility are associated with this design alternative. Additionally, this option is not as widely deployed in the field as the L2/L3 distribution layer boundary model. Distribution Layer Tuning The following are the recommendations for optimal distribution layer convergence: • Use equal-cost redundant connections to the core for fastest convergence and to avoid black holes. While it is tempting to reduce cost by reducing links between the distribution nodes to the core in a partial mesh design, the complexity and convergence tradeoffs related to this design are ultimately far more expensive. • Connect distribution nodes to facilitate summarization and L2 VLANs spanning multiple access layer switches where required. Summarization is required to facilitate optimum EIGRP or OSPF convergence. If summarization is implemented at the distribution layer, the distribution nodes must be linked or routing black holes occur. Additionally, in a less than optimal design where VLANs span multiple access layer switches, the distribution nodes must be linked by an L2 connection. Otherwise, multiple convergence events can occur for a single failure and undesirable traffic paths are taken after the spanning tree converges. • Utilize GLBP/HSRP millisecond timers. Convergence around a link or node failure in the L2/L3 distribution boundary model depends on default gateway redundancy and failover. Millisecond timers can reliably be implemented to achieve sub-second (800 ms) convergence based on HSRP/GLBP failover. • Tune GLBP/HSRP preempt delay to avoid black holes. Ensure that the distribution node has connectivity to the core before it preempts its HSRP/GLBP standby peer so that traffic is not dropped while connectivity to the core is established. • Tune EtherChannel and CEF load balancing to ensure optimum utilization of redundant, equal-cost links. Campus Network for High Availability Design Guide OL-15734-01 3 Design Recommendations Summary Monitor redundant link utilization in the hierarchical model and take steps to tune both L2 (EtherChannel) and L3 (CEF) links to avoid under-utilization. Use L3 and L4 (UDP/TCP port) information as input to hashing algorithms. When you use EtherChannel interconnections, use L3 and L4 information to achieve optimum utilization. When you use L3 routed equal-cost redundant paths, vary the input to the CEF hashing algorithm to improve load distribution. Use the default L3 information for the core nodes and use L3 with L4 information for the distribution nodes. Core Layer Tuning For optimum core layer convergence, build triangles, not squares, to take advantage of equal-cost redundant paths for the best deterministic convergence. When considering core topologies, it is important to consider the benefits of topologies with point-to-point links. Link up/down topology changes can be propagated almost immediately to the underlying protocols. Topologies with redundant equal-cost load sharing links are the most deterministic and optimized for convergence measured in milliseconds. With topologies that rely on indirect notification and timer-based detection, convergence is non-deterministic and convergence is measured in seconds. Design Guidance Review This section summarizes the campus network design recommendations, and includes the following topics: • Layer 3 Foundations Services, page 4 • Layer 2 Foundation Services, page 5 • General Design Considerations, page 6 Layer 3 Foundations Services The following are the design recommendations for Layer 3 foundation services: • Design for deterministic convergence—triangles, not squares. Topologies where point-to-point physical links are deployed provide the most deterministic convergence. Physical link up/down is faster than timer-based convergence. • Control peering across access layer links (passive interfaces). Unless you control L3 peering in the hierarchical campus model, the distribution nodes establish L3 peer relationships many times using the access nodes that they support, wasting memory and bandwidth. • Summarize at the distribution. It is important to summarize routing information as it leaves the distribution nodes towards the core for both EIGRP and OSPF. When you force summarization at this layer of the network, bounds are implemented on EIGRP queries and OSPF LSA/SPF propagation, which optimizes both routing protocols for campus convergence. • Optimize CEF for best utilization of redundant L3 paths. Campus Network for High Availability Design Guide 4 OL-15734-01 Design Recommendations Summary The hierarchical campus model implements many L3 equal-cost redundant paths. Typical traffic flows in the campus cross multiple redundant paths as traffic flows from the access layer across the distribution and core and into the data center. Unless you vary the decision input for the CEF hashing algorithm at the core and distribution layers, CEF polarization can result in under-utilization of redundant paths. Layer 2 Foundation Services The following are the design recommendations for Layer 2 foundation services: • Use Rapid PVST+ if you must span VLANs. If you are compelled by application requirements to depend on STP to resolve convergence events, use Rapid PVST+, which is far superior to 802.1d and even PVST+ (802.1d plus Cisco enhancements) from the convergence perspective. • Use Rapid PVST+ to protect against user-side loops. Even though the recommended design does not depend on STP to resolve link or node failure events, STP is required to protect against user-side loops. There are many ways that a loop can be introduced on the user-facing access layer ports. Wiring mistakes, misconfigured end stations, or malicious users can create a loop. STP is required to ensure a loop-free topology and to protect the rest of the network from problems created in the access layer. • Use the Spanning-Tree toolkit to protect against unexpected STP participation. Switches or workstations running a version of STP are commonly introduced into a network. This is not always a problem, such as when a switch is connected in a conference room to temporarily provide additional ports/connectivity. Sometimes this is undesirable, such as when the switch that is added has been configured to become the STP root for the VLANs to which it is attached. BDPU Guard and Root Guard are tools that can protect against these situations. BDPU Guard requires operator intervention if an unauthorized switch is connected to the network, and Root Guard protects against a switch configured in a way that would cause STP to converge when being connected to the network. • Use UDLD to protect against one-way up/up connections. In fiber topologies where fiber optic interconnections are used, which is common in a campus environment, physical misconnections can occur that allow a link to appear to be up/up when there is a mismatched set of transmit/receive pairs. When such a physical misconfiguration occurs, protocols such as STP can cause network instability. UDLD detects these physical misconfigurations and disables the ports in question. • Set trunks to on/on with no negotiate, prune unused VLANs, and use VTP transparent mode. When you configure switch-to-switch interconnections to carry multiple VLANs, set DTP to on/on with no negotiate to avoid DTP protocol negotiation. This tuning can save seconds of outage when restoring a failed link or node. Unused VLANs should be manually pruned from trunked interfaces to avoid broadcast propagation. Finally, VTP transparent mode should be used because the need for a shared VLAN database is lessened given current hierarchical network design. • Match PAgP settings between CatOS and Cisco IOS software. When connecting a Cisco IOS software device to a CatOS device, make sure that PAgP settings are the same on both sides. The defaults are different. CatOS devices should have PAgP set to off when connecting to a Cisco IOS software device if EtherChannels are not configured. Campus Network for High Availability Design Guide OL-15734-01 5 Design Recommendations Summary General Design Considerations The following are general design considerations: • Use HSRP or GLBP for default gateway redundancy (sub-second timers). Default gateway redundancy is an important component in convergence in a hierarchical network design. You can reliably tune HSRP/GLBP timers to achieve 900 ms convergence for link/node failure in the L2/L3 boundary in the distribution hierarchical model. • Deploy QoS end-to-end; protect the good and punish the bad. QoS is not just for voice and video anymore. Internet worms and denial of service (DoS) attacks have the ability to flood links even in a high-speed campus environment. You can use QoS policies to protect mission-critical applications while giving a lower class of service to suspect traffic. • Avoid daisy chaining stackable switches; stacks are good, StackWise and chassis solutions are better. Daisy-chained fixed configuration implementations add complexity. Without careful consideration, discontinuous VLAN/subnets, routing black holes, and active/active HSRP/GLPB situations can exist. Use StackWise technology in the Cisco Catalyst 3750 family or modular chassis implementations to avoid these complications. • Avoid asymmetric routing and unicast flooding; do not span VLANs across the access layer. When a less-than-optimal topology is used, a long-existing but frequently misunderstood situation can occur as a result of the difference between ARP and CAM table aging timers. If VLANs span across multiple access layer switches, return path traffic can be flooded to all access layer switches and end points. This can be easily avoided by not spanning VLANs across access layer switches. If this cannot be avoided, then tune the ARP aging timer so that it is less than the CAM aging timer. • Keep redundancy simple. Protecting against double failures by using three redundant links or three redundant nodes in the hierarchical design does not increase availability. Instead, it decreases availability by reducing serviceability and determinism. • Only span VLANs across multiple access layer switches if you must. Throughout this document we have discussed the challenges with an environment in which VLANs span access layer switches. This design is less than optimal from a convergence perspective. If you follow the rules, you can achieve deterministic convergence. However, there are many opportunities to increase your availability and optimize convergence with alternative designs. • L2/L3 distribution with HSRP or GLBP is a tried-and-true design. A network design that follows the tried-and-true topology in which the L2/L3 boundary is in the distribution layer is the most deterministic and can deliver sub-second (900 ms) convergence. When properly configured and tuned, this design is the recommended best practice. • L3 in the access is an emerging and intriguing option. Advances in routing protocols and campus hardware have made it viable to deploy a routing protocol in the access layer switches and use an L3 point-to-point routed link between the access and distribution layer switches. This design can provide improvement in several areas, most notably reliable convergence in the 60–200 ms range. Campus Network for High Availability Design Guide 6 OL-15734-01