In this post, I will provide the steps to configure Cisco Management Network Switches with multiple uplinks for redundant connectivity for Oracle Exadata Machines.
The initial configuration and deployment of an Oracle Exadata environment is performed using the Oracle Exadata Deployment Assistant (OEDA). Oracle Exadata requires a minimum of three networks: the Administration Network, the Client Network, and the Private Network.
What is Administration Network?:
Administration Network: Also known as the management network, this required network connects to your existing management network infrastructure, and is used for administrative work on all components of Oracle Exadata. By default, the administration network connects the database servers, storage servers, server Integrated Lights Out Manager (ILOM) interfaces, and RDMA Network Fabric switches to the Management Network Switch in the rack. One uplink is required from the Management Network Switch to your management network.
In general, the Exadata Management Network Switch is configured with a single uplink to the management network. This network is dedicated solely to management purposes by the DBA team, and no database traffic flows through it between application servers and database servers. In case of a failure or emergency, the network team can physically connect the switch to another port with ease. For this reason, the connection is often left with only one uplink and redundancy is typically ignored.
We decided to configure it with redundant connectivity. Although Oracle’s official documentation is generally excellent, in this case, we could not find a useful guide in either the Installation and Configuration Guide for Exadata Database Machine or on My Oracle Support.
“In the official documentation (Installation and Configuration Guide for Exadata Database Machine, section 6.4. – Configuring the Management Network Switch), it is stated that:
Note that the Cisco 4948E-F switch supports multiple uplinks to the customer network by utilizing ports 49 – 52. This is a more complicated switch setup due to the redundant connectivity, and should be performed by the customer’s network administrator.
While this might be a trivial task for a network administrator, I wanted to share all the steps, as they may be helpful to others. In this example, I will outline the steps for an Oracle Exadata X7-2 Half Rack with a Cisco 9348 Ethernet Switch, which has 22 available and currently unused ports.
Initial State:
exasw-adm# sh ver Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (C) 2002-2018, Cisco and/or its affiliates. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under their own licenses, such as open source. This software is provided "as is," and unless otherwise stated, there is no warranty, express or implied, including but not limited to warranties of merchantability and fitness for a particular purpose. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or GNU General Public License (GPL) version 3.0 or the GNU Lesser General Public License (LGPL) Version 2.1 or Lesser General Public License (LGPL) Version 2.0. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://opensource.org/licenses/gpl-3.0.html and http://www.opensource.org/licenses/lgpl-2.1.php and http://www.gnu.org/licenses/old-licenses/library.txt. Software BIOS: version 05.28 NXOS: version 7.0(3)I7(3) BIOS compile time: 01/18/2018 NXOS image file is: bootflash:///nxos.7.0.3.I7.3.bin NXOS compile time: 2/12/2018 13:00:00 [02/12/2018 22:13:48] Hardware cisco Nexus9000 C9348GC-FXP Chassis Intel(R) Xeon(R) CPU D-1526 @ 1.80GHz with 24571660 kB of memory. Processor Board ID FDO23020MYA Device name: exasw-adm bootflash: 115809095 kB Kernel uptime is 2169 day(s), 23 hour(s), 18 minute(s), 13 second(s) Last reset Reason: Unknown System version: 7.0(3)I7(3) Service: plugin Core Plugin, Ethernet Plugin Active Package(s):exasw-adm# sh int status -------------------------------------------------------------------------------- Port Name Status Vlan Duplex Speed Type -------------------------------------------------------------------------------- mgmt0 -- notconnec routed auto auto -- -------------------------------------------------------------------------------- Port Name Status Vlan Duplex Speed Type -------------------------------------------------------------------------------- Eth1/1 -- notconnec 1 auto auto 1g Eth1/2 -- notconnec 1 auto auto 1g Eth1/3 -- notconnec 1 auto auto 1g Eth1/4 -- notconnec 1 auto auto 1g Eth1/5 -- notconnec 1 auto auto 1g Eth1/6 -- notconnec 1 auto auto 1g Eth1/7 -- notconnec 1 auto auto 1g Eth1/8 -- notconnec 1 auto auto 1g Eth1/9 -- notconnec 1 auto auto 1g Eth1/10 -- notconnec 1 auto auto 1g Eth1/11 -- notconnec 1 auto auto 1g Eth1/12 -- notconnec 1 auto auto 1g Eth1/13 -- notconnec 1 auto auto 1g Eth1/14 -- notconnec 1 auto auto 1g Eth1/15 -- notconnec 1 auto auto 1g Eth1/16 -- notconnec 1 auto auto 1g Eth1/17 -- notconnec 1 auto auto 1g Eth1/18 -- notconnec 1 auto auto 1g Eth1/19 -- notconnec 1 auto auto 1g Eth1/20 -- notconnec 1 auto auto 1g Eth1/21 -- notconnec 1 auto auto 1g Eth1/22 -- notconnec 1 auto auto 1g Eth1/23 -- connected 1 full 1000 1g Eth1/24 -- connected 1 full 1000 1g Eth1/25 -- connected 1 full 1000 1g Eth1/26 -- connected 1 full 1000 1g Eth1/27 -- connected 1 full 1000 1g Eth1/28 -- connected 1 full 1000 1g Eth1/29 -- connected 1 full 1000 1g Eth1/30 -- connected 1 full 1000 1g Eth1/31 -- connected 1 full 1000 1g Eth1/32 -- connected 1 full 1000 1g Eth1/33 -- connected 1 full 1000 1g Eth1/34 -- connected 1 full 1000 1g Eth1/35 -- connected 1 full 1000 1g Eth1/36 -- connected 1 full 1000 1g Eth1/37 -- connected 1 full 1000 1g Eth1/38 -- connected 1 full 1000 1g Eth1/39 -- connected 1 full 1000 1g Eth1/40 -- connected 1 full 1000 1g Eth1/41 -- connected 1 full 1000 1g Eth1/42 -- connected 1 full 1000 1g Eth1/43 -- connected 1 full 1000 1g Eth1/44 -- connected 1 full 1000 1g Eth1/45 -- connected 1 full 1000 1g Eth1/46 -- connected 1 full 1000 1g Eth1/47 -- notconnec 1 auto auto 1g Eth1/48 -- connected 1 full 1000 1g Eth1/49 -- xcvrAbsen 1 auto auto -- Eth1/50 -- xcvrAbsen 1 auto auto -- Eth1/51 -- xcvrAbsen 1 auto auto -- Eth1/52 -- xcvrAbsen 1 auto auto -- Eth1/53 -- xcvrAbsen 1 auto auto -- Eth1/54 -- xcvrAbsen 1 auto auto -- Vlan1 -- connected routed auto auto -- exasw-adm# sh ip int br IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan1 XXX.XXX.XXX.XXX protocol-up/link-up/admin-upexasw-adm# sh port-channel summaryexasw-adm# sh vlan VLAN Name Status Ports ---- -------------------------------- --------- ------------------------------- 1 default active Eth1/1, Eth1/2, Eth1/3, Eth1/4 Eth1/5, Eth1/6, Eth1/7, Eth1/8 Eth1/9, Eth1/10, Eth1/11 Eth1/12, Eth1/13, Eth1/14 Eth1/15, Eth1/16, Eth1/17 Eth1/18, Eth1/19, Eth1/20 Eth1/21, Eth1/22, Eth1/23 Eth1/24, Eth1/25, Eth1/26 Eth1/27, Eth1/28, Eth1/29 Eth1/30, Eth1/31, Eth1/32 Eth1/33, Eth1/34, Eth1/35 Eth1/36, Eth1/37, Eth1/38 Eth1/39, Eth1/40, Eth1/41 Eth1/42, Eth1/43, Eth1/44 Eth1/45, Eth1/46, Eth1/47 Eth1/48, Eth1/49, Eth1/50 Eth1/51, Eth1/52, Eth1/53 Eth1/54 VLAN Type Vlan-mode ---- ----- ---------- 1 enet CE Remote SPAN VLANs ------------------------------------------------------------------------------- Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------exasw-adm# configure termexasw-adm#(config) feature ? analytics Enable/Disable Analytics!!! bash-shell Enable/Disable bash-shell bfd Bfd bgp Enable/Disable Border Gateway Protocol (BGP) catena Enable/Disable catena container-tracker Enable/Disable NXOS Container Tracker dhcp Enable/Disable DHCP Manager dot1x Enable/Disable dot1x eigrp Enable/Disable Enhanced Interior Gateway Routing Protocol (EIGRP) evb Enable/Disable Edge Virtual Bridge (EVB) evmed Enable/Disable Generic event detectors fabric Enable/Disable Fabric Services grpc Enable/Disable grpc Services hsrp Enable/Disable Hot Standby Router Protocol (HSRP) icam Enable/Disable icam imp Enable/Disable IMP interface-vlan Enable/Disable interface vlan isis Enable/Disable IS-IS Unicast Routing Protocol (IS-IS) itd Enable/Disable ITD lacp Enable/Disable LACP ldap Enable/Disable ldap lldp Enable/Disable LLDP msdp Enable/Disable Multicast Source Discovery Protocol (MSDP) nat Enable/Disable NAT nbm Enable/Disable Non Blocking Multicast (NBM) feature netconf Enable/Disable netconf Services netflow Enable/Disable NetFlow ngmvpn Enable/Disable EVPN/MVPN features ngoam Enable/Disable ngoam ntp Enable/Disable NTP nv Enable/Disable VxLAN nxapi Enable/Disable nxapi nxsdk Enable/Disable nxsdk Services openflow Enable/Disable OpenFlow agent ospf Enable/Disable Open Shortest Path First Protocol (OSPF) ospfv3 Enable/Disable Open Shortest Path First Version 3 Protocol (OSPFv3) password Credential(s) for the user(s)/device(s) pbr Enable/Disable Policy Based Routing(PBR) pim Enable/Disable Protocol Independent Multicast (PIM) pim6 Enable/Disable Protocol Independent Multicast (PIM) for IPv6
Ports 1 and 2 will be configured as uplink ports. Before physically connecting the cables to the backbone network, I will first complete the configuration and then proceed with the cabling.
LACP – Port Channel Configuration for multiple uplinks :
exasw-adm# configure terminalexasw-adm(config)# feature lacpexasw-adm(config)# int eth 1/1-2exasw-adm(config)# int eth 1/1-2exasw-adm(config-if-range)# channel-group 1 mode activeexasw-adm(config-if-range)# description *** MANAGEMENT_UPLINKexasw-adm(config-if-range)# int eth 1/1exasw-adm(config-if)# description Backbone1-Mgmt_Eth10/47exasw-adm(config-if)# int eth 1/2exasw-adm(config-if)# description Backbone2-Mgmt_Eth10/47
After completing the configuration, the next step is to establish the physical Ethernet connection and verify the interface status.
exasw-adm(config-if)# sh int status -------------------------------------------------------------------------------- Port Name Status Vlan Duplex Speed Type -------------------------------------------------------------------------------- mgmt0 -- notconnec routed auto auto -- -------------------------------------------------------------------------------- Port Name Status Vlan Duplex Speed Type -------------------------------------------------------------------------------- Eth1/1 Backbone1-Mgmt_Eth connected 1 full 1000 1g Eth1/2 Backbone2-Mgmt_Eth connected 1 full 1000 1g Eth1/3 -- notconnec 1 auto auto 1g Eth1/4 -- notconnec 1 auto auto 1g Eth1/5 -- notconnec 1 auto auto 1g Eth1/6 -- notconnec 1 auto auto 1g Eth1/7 -- notconnec 1 auto auto 1g Eth1/8 -- notconnec 1 auto auto 1g Eth1/9 -- notconnec 1 auto auto 1g Eth1/10 -- notconnec 1 auto auto 1g Eth1/11 -- notconnec 1 auto auto 1g Eth1/12 -- notconnec 1 auto auto 1g Eth1/13 -- notconnec 1 auto auto 1g Eth1/14 -- notconnec 1 auto auto 1g Eth1/15 -- notconnec 1 auto auto 1g Eth1/16 -- notconnec 1 auto auto 1g Eth1/17 -- notconnec 1 auto auto 1g Eth1/18 -- notconnec 1 auto auto 1g Eth1/19 -- notconnec 1 auto auto 1g Eth1/20 -- notconnec 1 auto auto 1g Eth1/21 -- notconnec 1 auto auto 1g Eth1/22 -- notconnec 1 auto auto 1g Eth1/23 -- connected 1 full 1000 1g Eth1/24 -- connected 1 full 1000 1g Eth1/25 -- connected 1 full 1000 1g Eth1/26 -- connected 1 full 1000 1g Eth1/27 -- connected 1 full 1000 1g Eth1/28 -- connected 1 full 1000 1g Eth1/29 -- connected 1 full 1000 1g Eth1/30 -- connected 1 full 1000 1g Eth1/31 -- connected 1 full 1000 1g Eth1/32 -- connected 1 full 1000 1g Eth1/33 -- connected 1 full 1000 1g Eth1/34 -- connected 1 full 1000 1g Eth1/35 -- connected 1 full 1000 1g Eth1/36 -- connected 1 full 1000 1g Eth1/37 -- connected 1 full 1000 1g Eth1/38 -- connected 1 full 1000 1g Eth1/39 -- connected 1 full 1000 1g Eth1/40 -- connected 1 full 1000 1g Eth1/41 -- connected 1 full 1000 1g Eth1/42 -- connected 1 full 1000 1g Eth1/43 -- connected 1 full 1000 1g Eth1/44 -- connected 1 full 1000 1g Eth1/45 -- connected 1 full 1000 1g Eth1/46 -- connected 1 full 1000 1g Eth1/47 -- notconnec 1 auto auto 1g Eth1/48 -- notconnec 1 auto auto 1g Eth1/49 -- xcvrAbsen 1 auto auto -- Eth1/50 -- xcvrAbsen 1 auto auto -- Eth1/51 -- xcvrAbsen 1 auto auto -- Eth1/52 -- xcvrAbsen 1 auto auto -- Eth1/53 -- xcvrAbsen 1 auto auto -- Eth1/54 -- xcvrAbsen 1 auto auto -- Po1 -- connected 1 full 1000 -- Vlan1 -- connected routed auto auto --exasw-adm(config-if)# end
Check the spanning tree configuration to validate the root interface.
exasw-adm# show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 16412 Address 7683.efff.bd58 Cost 3 Port 4096 (port-channel1) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32769 (priority 32768 sys-id-ext 1) Address d4e8.8025.bf35 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po1 Root FWD 3 128.4096 P2p Eth1/23 Desg FWD 4 128.89 Edge P2p Eth1/24 Desg FWD 4 128.93 Edge P2p Eth1/25 Desg FWD 4 128.97 Edge P2p Eth1/26 Desg FWD 4 128.101 Edge P2p Eth1/27 Desg FWD 4 128.105 Edge P2p Eth1/28 Desg FWD 4 128.109 Edge P2p Eth1/29 Desg FWD 4 128.113 Edge P2p Eth1/30 Desg FWD 4 128.117 Edge P2p Eth1/31 Desg FWD 4 128.121 Edge P2p Eth1/32 Desg FWD 4 128.125 Edge P2p Eth1/33 Desg FWD 4 128.129 Edge P2p Eth1/34 Desg FWD 4 128.133 Edge P2p Eth1/35 Desg FWD 4 128.137 Edge P2p Eth1/36 Desg FWD 4 128.141 Edge P2p Eth1/37 Desg FWD 4 128.145 Edge P2p Eth1/38 Desg FWD 4 128.149 Edge P2p Eth1/39 Desg FWD 4 128.153 Edge P2p Eth1/40 Desg FWD 4 128.157 Edge P2p Eth1/41 Desg FWD 4 128.161 Edge P2p Eth1/42 Desg FWD 4 128.165 Edge P2p Eth1/43 Desg FWD 4 128.169 Edge P2p Eth1/44 Desg FWD 4 128.173 Edge P2p Eth1/45 Desg FWD 4 128.177 Edge P2p Eth1/46 Desg FWD 4 128.181 Edge P2p
Use the copy running-config startup-config command to save the current configuration and make it persistent across reboots.
exasw-adm# copy running-config startup-config [########################################] 100% Copy complete, now saving to disk (please wait)... Copy complete.
Hope it helps.
Leave your comment