In this post, I will provide the steps to configure Cisco Management Network Switches with multiple uplinks for redundant connectivity for Oracle Exadata Machines.
The initial configuration and deployment of an Oracle Exadata environment is performed using the Oracle Exadata Deployment Assistant (OEDA). Oracle Exadata requires a minimum of three networks: the Administration Network, the Client Network, and the Private Network.
What is Administration Network?:
Administration Network: Also known as the management network, this required network connects to your existing management network infrastructure, and is used for administrative work on all components of Oracle Exadata. By default, the administration network connects the database servers, storage servers, server Integrated Lights Out Manager (ILOM) interfaces, and RDMA Network Fabric switches to the Management Network Switch in the rack. One uplink is required from the Management Network Switch to your management network.
In general, the Exadata Management Network Switch is configured with a single uplink to the management network. This network is dedicated solely to management purposes by the DBA team, and no database traffic flows through it between application servers and database servers. In case of a failure or emergency, the network team can physically connect the switch to another port with ease. For this reason, the connection is often left with only one uplink and redundancy is typically ignored.
We decided to configure it with redundant connectivity. Although Oracle’s official documentation is generally excellent, in this case, we could not find a useful guide in either the Installation and Configuration Guide for Exadata Database Machine or on My Oracle Support.
“In the official documentation (Installation and Configuration Guide for Exadata Database Machine, section 6.4. – Configuring the Management Network Switch), it is stated that:
Note that the Cisco 4948E-F switch supports multiple uplinks to the customer network by utilizing ports 49 – 52. This is a more complicated switch setup due to the redundant connectivity, and should be performed by the customer’s network administrator.
While this might be a trivial task for a network administrator, I wanted to share all the steps, as they may be helpful to others. In this example, I will outline the steps for an Oracle Exadata X7-2 Half Rack with a Cisco 9348 Ethernet Switch, which has 22 available and currently unused ports.
Initial State:
exasw-adm# sh ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2018, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source. This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0 or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
Software
BIOS: version 05.28
NXOS: version 7.0(3)I7(3)
BIOS compile time: 01/18/2018
NXOS image file is: bootflash:///nxos.7.0.3.I7.3.bin
NXOS compile time: 2/12/2018 13:00:00 [02/12/2018 22:13:48]
Hardware
cisco Nexus9000 C9348GC-FXP Chassis
Intel(R) Xeon(R) CPU D-1526 @ 1.80GHz with 24571660 kB of memory.
Processor Board ID FDO23020MYA
Device name: exasw-adm
bootflash: 115809095 kB
Kernel uptime is 2169 day(s), 23 hour(s), 18 minute(s), 13 second(s)
Last reset
Reason: Unknown
System version: 7.0(3)I7(3)
Service:
plugin
Core Plugin, Ethernet Plugin
Active Package(s):
exasw-adm# sh int status
--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
mgmt0 -- notconnec routed auto auto --
--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 -- notconnec 1 auto auto 1g
Eth1/2 -- notconnec 1 auto auto 1g
Eth1/3 -- notconnec 1 auto auto 1g
Eth1/4 -- notconnec 1 auto auto 1g
Eth1/5 -- notconnec 1 auto auto 1g
Eth1/6 -- notconnec 1 auto auto 1g
Eth1/7 -- notconnec 1 auto auto 1g
Eth1/8 -- notconnec 1 auto auto 1g
Eth1/9 -- notconnec 1 auto auto 1g
Eth1/10 -- notconnec 1 auto auto 1g
Eth1/11 -- notconnec 1 auto auto 1g
Eth1/12 -- notconnec 1 auto auto 1g
Eth1/13 -- notconnec 1 auto auto 1g
Eth1/14 -- notconnec 1 auto auto 1g
Eth1/15 -- notconnec 1 auto auto 1g
Eth1/16 -- notconnec 1 auto auto 1g
Eth1/17 -- notconnec 1 auto auto 1g
Eth1/18 -- notconnec 1 auto auto 1g
Eth1/19 -- notconnec 1 auto auto 1g
Eth1/20 -- notconnec 1 auto auto 1g
Eth1/21 -- notconnec 1 auto auto 1g
Eth1/22 -- notconnec 1 auto auto 1g
Eth1/23 -- connected 1 full 1000 1g
Eth1/24 -- connected 1 full 1000 1g
Eth1/25 -- connected 1 full 1000 1g
Eth1/26 -- connected 1 full 1000 1g
Eth1/27 -- connected 1 full 1000 1g
Eth1/28 -- connected 1 full 1000 1g
Eth1/29 -- connected 1 full 1000 1g
Eth1/30 -- connected 1 full 1000 1g
Eth1/31 -- connected 1 full 1000 1g
Eth1/32 -- connected 1 full 1000 1g
Eth1/33 -- connected 1 full 1000 1g
Eth1/34 -- connected 1 full 1000 1g
Eth1/35 -- connected 1 full 1000 1g
Eth1/36 -- connected 1 full 1000 1g
Eth1/37 -- connected 1 full 1000 1g
Eth1/38 -- connected 1 full 1000 1g
Eth1/39 -- connected 1 full 1000 1g
Eth1/40 -- connected 1 full 1000 1g
Eth1/41 -- connected 1 full 1000 1g
Eth1/42 -- connected 1 full 1000 1g
Eth1/43 -- connected 1 full 1000 1g
Eth1/44 -- connected 1 full 1000 1g
Eth1/45 -- connected 1 full 1000 1g
Eth1/46 -- connected 1 full 1000 1g
Eth1/47 -- notconnec 1 auto auto 1g
Eth1/48 -- connected 1 full 1000 1g
Eth1/49 -- xcvrAbsen 1 auto auto --
Eth1/50 -- xcvrAbsen 1 auto auto --
Eth1/51 -- xcvrAbsen 1 auto auto --
Eth1/52 -- xcvrAbsen 1 auto auto --
Eth1/53 -- xcvrAbsen 1 auto auto --
Eth1/54 -- xcvrAbsen 1 auto auto --
Vlan1 -- connected routed auto auto --
exasw-adm# sh ip int br
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Vlan1 XXX.XXX.XXX.XXX protocol-up/link-up/admin-up
exasw-adm# sh port-channel summary
exasw-adm# sh vlan
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Eth1/1, Eth1/2, Eth1/3, Eth1/4
Eth1/5, Eth1/6, Eth1/7, Eth1/8
Eth1/9, Eth1/10, Eth1/11
Eth1/12, Eth1/13, Eth1/14
Eth1/15, Eth1/16, Eth1/17
Eth1/18, Eth1/19, Eth1/20
Eth1/21, Eth1/22, Eth1/23
Eth1/24, Eth1/25, Eth1/26
Eth1/27, Eth1/28, Eth1/29
Eth1/30, Eth1/31, Eth1/32
Eth1/33, Eth1/34, Eth1/35
Eth1/36, Eth1/37, Eth1/38
Eth1/39, Eth1/40, Eth1/41
Eth1/42, Eth1/43, Eth1/44
Eth1/45, Eth1/46, Eth1/47
Eth1/48, Eth1/49, Eth1/50
Eth1/51, Eth1/52, Eth1/53
Eth1/54
VLAN Type Vlan-mode
---- ----- ----------
1 enet CE
Remote SPAN VLANs
-------------------------------------------------------------------------------
Primary Secondary Type Ports
------- --------- --------------- -------------------------------------------
exasw-adm# configure term
exasw-adm#(config) feature ?
analytics Enable/Disable Analytics!!!
bash-shell Enable/Disable bash-shell
bfd Bfd
bgp Enable/Disable Border Gateway Protocol (BGP)
catena Enable/Disable catena
container-tracker Enable/Disable NXOS Container Tracker
dhcp Enable/Disable DHCP Manager
dot1x Enable/Disable dot1x
eigrp Enable/Disable Enhanced Interior Gateway Routing Protocol (EIGRP)
evb Enable/Disable Edge Virtual Bridge (EVB)
evmed Enable/Disable Generic event detectors
fabric Enable/Disable Fabric Services
grpc Enable/Disable grpc Services
hsrp Enable/Disable Hot Standby Router Protocol (HSRP)
icam Enable/Disable icam
imp Enable/Disable IMP
interface-vlan Enable/Disable interface vlan
isis Enable/Disable IS-IS Unicast Routing Protocol (IS-IS)
itd Enable/Disable ITD
lacp Enable/Disable LACP
ldap Enable/Disable ldap
lldp Enable/Disable LLDP
msdp Enable/Disable Multicast Source Discovery Protocol (MSDP)
nat Enable/Disable NAT
nbm Enable/Disable Non Blocking Multicast (NBM) feature
netconf Enable/Disable netconf Services
netflow Enable/Disable NetFlow
ngmvpn Enable/Disable EVPN/MVPN features
ngoam Enable/Disable ngoam
ntp Enable/Disable NTP
nv Enable/Disable VxLAN
nxapi Enable/Disable nxapi
nxsdk Enable/Disable nxsdk Services
openflow Enable/Disable OpenFlow agent
ospf Enable/Disable Open Shortest Path First Protocol (OSPF)
ospfv3 Enable/Disable Open Shortest Path First Version 3 Protocol (OSPFv3)
password Credential(s) for the user(s)/device(s)
pbr Enable/Disable Policy Based Routing(PBR)
pim Enable/Disable Protocol Independent Multicast (PIM)
pim6 Enable/Disable Protocol Independent Multicast (PIM) for IPv6
Ports 1 and 2 will be configured as uplink ports. Before physically connecting the cables to the backbone network, I will first complete the configuration and then proceed with the cabling.
LACP – Port Channel Configuration for multiple uplinks :
exasw-adm# configure terminal
exasw-adm(config)# feature lacp
exasw-adm(config)# int eth 1/1-2
exasw-adm(config)# int eth 1/1-2
exasw-adm(config-if-range)# channel-group 1 mode active
exasw-adm(config-if-range)# description *** MANAGEMENT_UPLINK
exasw-adm(config-if-range)# int eth 1/1
exasw-adm(config-if)# description Backbone1-Mgmt_Eth10/47
exasw-adm(config-if)# int eth 1/2
exasw-adm(config-if)# description Backbone2-Mgmt_Eth10/47After completing the configuration, the next step is to establish the physical Ethernet connection and verify the interface status.
exasw-adm(config-if)# sh int status
--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
mgmt0 -- notconnec routed auto auto --
--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 Backbone1-Mgmt_Eth connected 1 full 1000 1g
Eth1/2 Backbone2-Mgmt_Eth connected 1 full 1000 1g
Eth1/3 -- notconnec 1 auto auto 1g
Eth1/4 -- notconnec 1 auto auto 1g
Eth1/5 -- notconnec 1 auto auto 1g
Eth1/6 -- notconnec 1 auto auto 1g
Eth1/7 -- notconnec 1 auto auto 1g
Eth1/8 -- notconnec 1 auto auto 1g
Eth1/9 -- notconnec 1 auto auto 1g
Eth1/10 -- notconnec 1 auto auto 1g
Eth1/11 -- notconnec 1 auto auto 1g
Eth1/12 -- notconnec 1 auto auto 1g
Eth1/13 -- notconnec 1 auto auto 1g
Eth1/14 -- notconnec 1 auto auto 1g
Eth1/15 -- notconnec 1 auto auto 1g
Eth1/16 -- notconnec 1 auto auto 1g
Eth1/17 -- notconnec 1 auto auto 1g
Eth1/18 -- notconnec 1 auto auto 1g
Eth1/19 -- notconnec 1 auto auto 1g
Eth1/20 -- notconnec 1 auto auto 1g
Eth1/21 -- notconnec 1 auto auto 1g
Eth1/22 -- notconnec 1 auto auto 1g
Eth1/23 -- connected 1 full 1000 1g
Eth1/24 -- connected 1 full 1000 1g
Eth1/25 -- connected 1 full 1000 1g
Eth1/26 -- connected 1 full 1000 1g
Eth1/27 -- connected 1 full 1000 1g
Eth1/28 -- connected 1 full 1000 1g
Eth1/29 -- connected 1 full 1000 1g
Eth1/30 -- connected 1 full 1000 1g
Eth1/31 -- connected 1 full 1000 1g
Eth1/32 -- connected 1 full 1000 1g
Eth1/33 -- connected 1 full 1000 1g
Eth1/34 -- connected 1 full 1000 1g
Eth1/35 -- connected 1 full 1000 1g
Eth1/36 -- connected 1 full 1000 1g
Eth1/37 -- connected 1 full 1000 1g
Eth1/38 -- connected 1 full 1000 1g
Eth1/39 -- connected 1 full 1000 1g
Eth1/40 -- connected 1 full 1000 1g
Eth1/41 -- connected 1 full 1000 1g
Eth1/42 -- connected 1 full 1000 1g
Eth1/43 -- connected 1 full 1000 1g
Eth1/44 -- connected 1 full 1000 1g
Eth1/45 -- connected 1 full 1000 1g
Eth1/46 -- connected 1 full 1000 1g
Eth1/47 -- notconnec 1 auto auto 1g
Eth1/48 -- notconnec 1 auto auto 1g
Eth1/49 -- xcvrAbsen 1 auto auto --
Eth1/50 -- xcvrAbsen 1 auto auto --
Eth1/51 -- xcvrAbsen 1 auto auto --
Eth1/52 -- xcvrAbsen 1 auto auto --
Eth1/53 -- xcvrAbsen 1 auto auto --
Eth1/54 -- xcvrAbsen 1 auto auto --
Po1 -- connected 1 full 1000 --
Vlan1 -- connected routed auto auto --
exasw-adm(config-if)# endCheck the spanning tree configuration to validate the root interface.
exasw-adm# show spanning-tree
VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 16412
Address 7683.efff.bd58
Cost 3
Port 4096 (port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address d4e8.8025.bf35
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 3 128.4096 P2p
Eth1/23 Desg FWD 4 128.89 Edge P2p
Eth1/24 Desg FWD 4 128.93 Edge P2p
Eth1/25 Desg FWD 4 128.97 Edge P2p
Eth1/26 Desg FWD 4 128.101 Edge P2p
Eth1/27 Desg FWD 4 128.105 Edge P2p
Eth1/28 Desg FWD 4 128.109 Edge P2p
Eth1/29 Desg FWD 4 128.113 Edge P2p
Eth1/30 Desg FWD 4 128.117 Edge P2p
Eth1/31 Desg FWD 4 128.121 Edge P2p
Eth1/32 Desg FWD 4 128.125 Edge P2p
Eth1/33 Desg FWD 4 128.129 Edge P2p
Eth1/34 Desg FWD 4 128.133 Edge P2p
Eth1/35 Desg FWD 4 128.137 Edge P2p
Eth1/36 Desg FWD 4 128.141 Edge P2p
Eth1/37 Desg FWD 4 128.145 Edge P2p
Eth1/38 Desg FWD 4 128.149 Edge P2p
Eth1/39 Desg FWD 4 128.153 Edge P2p
Eth1/40 Desg FWD 4 128.157 Edge P2p
Eth1/41 Desg FWD 4 128.161 Edge P2p
Eth1/42 Desg FWD 4 128.165 Edge P2p
Eth1/43 Desg FWD 4 128.169 Edge P2p
Eth1/44 Desg FWD 4 128.173 Edge P2p
Eth1/45 Desg FWD 4 128.177 Edge P2p
Eth1/46 Desg FWD 4 128.181 Edge P2pUse the copy running-config startup-config command to save the current configuration and make it persistent across reboots.
exasw-adm# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete.Hope it helps.
Osman’s DBlog 
Leave your comment