N7k spanning-tree vlan

February 18, 2014, 9:20 pm

≫ Next: "SFP failed" issue in Nexus 5K ?

task:

TO assign a given switch to be the root node by setting its root priority:

config:

N7K-1(config-if)# spanning-tree vlan 100 priority 8192

fabricpath config:

N7K-1# show license usage
Feature Ins Lic Status Expiry Date Comments
Count
----------------------------------------------------------------------------
MPLS_PKG No - Unused -
STORAGE-ENT No - Unused -
ENTERPRISE_PKG No - Unused -
FCOE-N7K-F132XP No 0 Unused -
ENHANCED_LAYER2_PKG Yes - Unused Never -
SCALABLE_SERVICES_PKG No - Unused -
TRANSPORT_SERVICES_PKG Yes - Unused Never -
LAN_ADVANCED_SERVICES_PKG Yes - Unused Never -
LAN_ENTERPRISE_SERVICES_PKG Yes - Unused Never -
----------------------------------------------------------------------------

===config ===

===N7K====

feature-set fabricpath
vlan 100
mode fabricpath
int e1/11-18
switchport mode fabricpath
no shut

===N5K====

N5K-1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
N5K-1(config)# install feature-set fabricpath
N5K-1(config)# feature-set fabricpath
N5K-1(config)# vlan 100
N5K-1(config-vlan)# mode fabricpath
N5K-1(config)# int e1/1-8
N5K-1(config-if-range)# switchport mode fabricpath
N5K-1(config-if-range)# no shutdown

==verify===

show fabricpath isis interface brief

show fabricpath isis

====task:===
how to enable FabricPath on all F series module interfaces within a given VDC?
===config===
system default switchport fabricpath

====task:===
influence the root selection in fabricpath:
===config===
N7K-1(config)# fabricpath domain default
N7K-1(config-fabricpath-isis)# root-priority 255

↧

"SFP failed" issue in Nexus 5K ?

March 12, 2014, 7:40 pm

≫ Next: loop detection: mac move notification in n5k

≪ Previous: N7k spanning-tree vlan

1. check if your SFP is supported here
2. If you are using 1G SFP, check if you have configured the proper speed, the speed of interface by default is 10G. (you can check this by show run int eX/Y all)

↧

loop detection: mac move notification in n5k

March 12, 2014, 8:59 pm

≫ Next: nexus basic config

≪ Previous: "SFP failed" issue in Nexus 5K ?

This is one of my favorite features to enable in Nexus and IOS platform, it makes troubleshooting so much easier and logical.

==What is mac move notification? ===

as discussed here:

Usage Guidelines

MAC-move notification generates a syslog message whenever a MAC address or host moves between different switch ports.

MAC-move notification does not generate a notification when a new MAC address is added to the content-addressable memory (CAM) or when a MAC address is removed from the CAM.

MAC-move notification is supported on switch ports only.

The MAC-move counter notification generates a syslog message when the number of MAC moves in a VLAN exceeds the maximum limit. The maximum limit is 1000 MAC moves.

The MAC-move counter syslog notification counts the number of times a MAC has moved within a VLAN and the number of these instances that have occurred in the system.

====how to configure it ? ======

Examples

This example shows how to enable MAC-move notification:

Router(config)# mac-address-table notification mac-move

This example shows how to disable MAC-move notification:

Router(config)# no mac-address-table notification mac-move

This example shows how to enable MAC-move counter syslog notification:

Router(config)# mac-address-table notification mac-move counter syslog

This example shows how to disable MAC-move counter notification:

Router(config)# no mac-address-table notification mac-move counter

In Nexus :

Nexus-5000# conf t
Nexus-5000(config)# mac address-table notification mac-move
With Nexus 5000 switches, it is not always sufficient to enable the MAC-move notification in order to generate a syslog message about MAC-move notification.
In order to ensure syslog message generation, enter these commands in conjunction with the previous command.
Nexus-5000# conf t
Nexus-5000(config)# Logging level spanning-tree 6
Nexus-5000(config)# Logging level fwm 6
Nexus-5000(config)# Logging monitor 6

===Why mac move? ====

It is common to see mac flapping in L2 network, without enabling mac move, if there are mac flaps, you will see something like this:

2011 Nov 20 09:38:04.743 Prod-Sw-2a_13_113 %FWM-2-STM_LOOP_DETECT: Loops
detected in the network among ports Eth115/1/33 and Po1 vlan 815 -
Disabling dynamic learn notificationsfor 180 seconds
2011 Nov 20 09:41:04.782 Prod-Sw-2a_13_113
%FWM-2-STM_LEARNING_RE_ENABLE: Re enabling dynamic learning on all
interfaces
2011 Nov 20 09:41:05.231 Prod-Sw-2a_13_113 %FWM-2-STM_LOOP_DETECT: Loops
detected in the network among ports Po1 and Eth115/1/35 vlan 815 -
Disabling dynamic learn notificationsfor 180 seconds
2011 Nov 20 09:44:05.260 Prod-Sw-2a_13_113
%FWM-2-STM_LEARNING_RE_ENABLE: Re enabling dynamic learning on all
interfaces
2011 Nov 20 09:49:08.125 Prod-Sw-2a_13_113 %VSHD-5-VSHD_SYSLOG_CONFIG_I:
Configured from vty by asad on 172.17.161.0@pts/34
2011 Nov 20 10:00:32.475 Prod-Sw-2a_13_113 %FWM-2-STM_LOOP_DETECT: Loops
detected in the network among ports Po1 and Eth115/1/35 vlan 815 -
Disabling dynamic learn notificationsfor 180 seconds

after enabling mac move notification:

the log should looks like this :

2014 Feb 6 04:02:46.192 sw-a-4.or1 FWM-2-STM_LOOP_DETECT Loops detected in the network for mac 2c44.fd7c.654c among ports Eth105/1/28 and Eth105/1/19 vlan 1251 - Disabling dynamic learn notifications for 180 seconds
2014 Feb 6 04:05:46.509 sw-a-4.or1 FWM-2-STM_LOOP_DETECT Loops detected in the network for mac 2c44.fd7d.7690 among ports Eth105/1/20 and Eth105/1/28 vlan 1251 - Disabling dynamic learn notifications for 180 seconds
2014 Feb 6 04:08:47.297 sw-a-4.or1 FWM-2-STM_LOOP_DETECT Loops detected in the network for mac 2c44.fd7d.24ac among ports Eth105/1/2 and Eth105/1/28 vlan 1251 - Disabling dynamic learn notifications for 180 seconds
2014 Feb 6 04:11:48.197 sw-a-4.or1 FWM-2-STM_LOOP_DETECT Loops detected in the network for mac d89d.6728.e530 among ports gpc1:103 and Eth105/1/28 vlan 1251 - Disabling dynamic learn notifications for 180 seconds

Knowing what mac is moving will make tracking down the loop much easier.

note:

It is normal to see some mac move in a virtualized environment due to dynamic load balancing and Vmotion.

==very good read===

Nexus 5000 FAQ: What do you do when a Nexus 5000 switch displays the "FWM-2-STM_LOOP_DETECT" message in the log?

http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/116200-qanda-nexus5000-00.html

↧

nexus basic config

May 21, 2014, 4:33 am

≫ Next: How to remove and install the layer 3 daughter card

≪ Previous: loop detection: mac move notification in n5k

--change spanning-tree priority , setting STP root brige

conf t
spanning-tree vlan 100 priority 8192

fabricpath:

need enhanced l2 license

" show license usage"

==mtu==

in n7k, can configure mtu per interface:

int ##
mtu 9216

==priority ===

role priority vs system priority

both default values are 32667,
configurable,
if you are to change it,
need to make sure:

system priority must be the SAME on both Nexus switches.
if they are not, you will see vpc errors.

role priority priority

Example:

switch(config-vpc-domain)# role priority 4

switch(config-vpc-domain)#

Enters the role priority that you want for the vPC system priority.The range of values is from 1 to 65636, and the default value is 32667. A lower value means this switch has a better chance of being the primary vPC.

↧

How to remove and install the layer 3 daughter card

May 27, 2014, 6:15 pm

≫ Next: UCS LDAP raw note

≪ Previous: nexus basic config

if you RMA a N5K with L3 module, please be mindful that you may have some extra task to do.
one of it is to remove the L3 module from the old N5K switch to the new one.
here is how:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/hw/installation/guide/replacement.html#wp1059316

one thing to note is how the IO module looks like

↧

UCS LDAP raw note

August 14, 2014, 11:44 pm

≫ Next: setting up FC port channel between MDS and N5K, issues, errors and fix

≪ Previous: How to remove and install the layer 3 daughter card

if authentication to GUI failed after doing LDAP configuration on UCS, a good way to troubleshoot is from CLI:
use command:

connect nxos
test aaa group ## (group name) ## (username)## (password)

here will give more detail why it fails.

one common error is "bind fail", then we just need to check the Bind User configuration, make sure
" cn=" is included for the bind name.

** group-map:
is to enable the mapped group to have certain level of authorization. eg: network, aaa..

** group recursion:

when enabled, will allow search from root of the tree down to each level.

** port:
default is 389, no SSL

** why if group-map is deleted ?

by default ,user can still log in, but the authorization is " read-only".

↧

setting up FC port channel between MDS and N5K, issues, errors and fix

August 15, 2014, 5:38 am

≫ Next: MDS: zoneset distribute

≪ Previous: UCS LDAP raw note

problem:

some error messages when setting up a FC port channel between a MDS and N5K:
* switch(config-if)# 2014 Aug 15 10:46:31 switch %$ VDC-1 %$ %FCDOMAIN-2-EPORT_ISOLATED: %$VSAN 1000%$ Isolation of interface san-port-channel 10 (reason: domains are overlapping)

Topology:

N5K-----int san-po10-----------int po10----MDS (allow vsan 1000 to pass through)

Troubleshooting on N5K:
switch(config-if)# sh fcdomain domain-list vsan 1000

Number of domains: 1
Domain ID              WWN
---------    -----------------------
0xef(239)    ###### [Local] [Principal]
and on MDS:
switch(config-if)# sh fcdomain domain-list vsan 1000

Number of domains: 1
Domain ID              WWN
---------    -----------------------
0xef(239)    ######[Local] [Principal]

ok, I can see the overlapping here, so i first try to configure it in a gentle way, the " preferred " one:
(on both sides )

switch(config-if)# fcdomain domain 238 preferred vsan 1000
switch(config)#
switch(config)#

switch(config)# fcdomain restart vsan 1000

sadly, it doesn't make a difference.

===edited ====
after reading this blog: it seems like I need to add the HIDDEN command: "disruptive", will test this out again in the lab later:

======from the blog above=====

If you say preferred, the switch will accept any FCID but prefers the domain-id you specify.

fcdomain domain 10 preferred vsan 10

if you are on a switch, and you want to have this actually take affect you need to do a restart, more importantly for preferred fcdomain's you need to do a DISRUPTIVE restart:

fcdomain restart disruptive vsan 10

the DISRUPTIVE keyword is hidden, keep that in mind!

==============================
switch(config)# sh fcdomain domain-list vsan 1000

Number of domains: 1
Domain ID              WWN
---------    -----------------------
0xef(239)    ##Local] [Principal]

now I got another error:

switch(config)# 2014 Aug 15 10:56:32 switch %$ VDC-1 %$ %FCDOMAIN-2-EPORT_ISOLATED: %$VSAN 1000%$ Isolation of interface san-port-channel 10 (reason: configured domain ID is different from runtime domain ID)

further research on cco here: points me to another thing to try: (I did it on both MDS and N5K)

switch(config)# fcdomain auto-reconfigure vsan 1000
switch(config)# fcdomain restart vsan 1000
switch(config)#
switch(config)#
switch(config)#
switch(config)# sh fcdomain domain-list vsan 1000

Number of domains: 1
Domain ID              WWN
---------    -----------------------
0xef(239)   ###[Local] [Principal]

the domain ID is still the same old one. :(

finally, I have to try something more brutal, the " static" domain ID:

switch(config)# fcdomain domain 236 static vsan 1000 (configure a different value on the other side )
switch(config)#
switch(config)#
switch(config)# fcdomain restart vsan 1000
switch(config)#
switch(config)#
switch(config)#
switch(config)#
switch(config)# sh fcdomain domain-list vsan 1000

with this, my san-port-channel come up eventually after another flapping :

n5k (config-if)# sh int san-port-channel 10
san-port-channel 10 is trunking
    Hardware is Fibre Channel
    Port WWN is ###
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Speed is 8 Gbps
    Trunk vsans (admin allowed and active) (1000)
    Trunk vsans (up)                       (1000)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             ()
    1 minute input rate 3000 bits/sec, 375 bytes/sec, 4 frames/sec
    1 minute output rate 2912 bits/sec, 364 bytes/sec, 4 frames/sec
      559 frames input, 59248 bytes
        0 discards, 0 errors
        0 CRC, 0 unknown class
        0 too long, 0 too short
      561 frames output, 47936 bytes
        0 discards, 0 errors
      0 input OLS, 14 LRR, 0 NOS, 0 loop inits
      17 output OLS, 16 LRR, 10 NOS, 0 loop inits
    last clearing of "show interface" counters never
    Member[1] : fc2/1
    Member[2] : fc2/2
    Interface last changed at Fri Aug 15 11:06:08 2014

note: pwwn and wwn are not shown here for obvious reason. :)

further reference: very neat troubleshooting guide: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/N5K_Troubleshooting_Guide/n5K_ts_sans.html#wp1045492

http://www.cisco.com/en/US/products/ps5989/prod_troubleshooting_guide_chapter09186a008067a306.html

http://www.ccierants.com/2013/06/ccie-dc-advanced-fcoe.html

↧

MDS: zoneset distribute

August 15, 2014, 11:29 pm

≫ Next: npv traffic-map

≪ Previous: setting up FC port channel between MDS and N5K, issues, errors and fix

Some people are confused about when and why we need to do "zoneset distribute" in MDS,
here is the answer:

•

If using basic zoning, then choose Fabricxx > VSANxx > zonesetname and select FullZoneSet from the Propagation drop-down menu in Fabric Manager.

Or use the zoneset distribute full vsan CLI command to distribute the full zone database across the fabric whenever a zone set activation occurs. This ensures a consistent full zone database on all switches for that VSAN.

from CCO guide.

How do I know if I am running basic zoning?

from the same guide, you can find the information below:

Step 1

Issue the show zone status command.

v_188# show zone status

VSAN: 1 default-zone: deny distribute: active only Interop: default

mode: basic merge-control: allow session: none  <------------------

    hard-zoning: enabled

Default zone:

    qos: low broadcast: disabled ronly: disabled

Full Zoning Database :

    Zonesets:5  Zones:18 Aliases: 11

Active Zoning Database :

    Name: ZoneSet1  Zonesets:1  Zones:2

Status:

This example shows the default zone policy is deny, and the zone mode is basic.

==Sample configuration ====

here is some sample configuration for zoning:

vsan database
vsan ###

zone name zone_1 vsan ###
member pwwn ###
member pwwn ###

.....

zoneset name zs_1 vsan 100
member zone_1

zoneset activate name zs_1 vsan 100

zoneset distribute full vsan 100

↧

npv traffic-map

August 16, 2014, 3:42 am

≫ Next: FCoE raw note

≪ Previous: MDS: zoneset distribute

I come across this concept and just putting some notes here to remind myself:

this is some sort of traffic engineering.
======N5K=======
1. good old trusty cco guide:

Configuring NPV Traffic Maps

An NPV traffic map associates one or more NP uplink interfaces with a server interface. The switch associates the server interface with one of these NP uplinks.

Note

If a server interface is already mapped to an NP uplink, you should include this mapping in the traffic map configuration.

To configure a traffic map, perform this task:


	Command	Purpose
Step 1	switch# config t switch(config)#	Enters configuration mode on the NPV.
Step 2	switch(config)# npv traffic-map server-interface {fc slot/port \| vfc vfc-id} external-interface fc slot/port switch (config)#	Configures a mapping between a server interface (or range of server interfaces) and an NP uplink interface (or range of NP uplink interfaces).
Step 2	switch(config)# no npv traffic-map server-interface {fc slot/port \| vfc vfc-id}external-interface fc slot/port switch (config)#	Removes the mapping between the specified server interfaces and NP uplink interfaces.

Note

The traffic map configuration only takes effect after you reinitialize each of the server interfaces specified in the map.

Command Default

No traffic map. The switch uses automatic uplink selection to select an NP uplink for the server interface.

http://www.cisco.com/web/techdoc/dc/reference/cli/nxos/commands/fc/npv_traffic-map.html

Usage Guidelines

This command is only available when the switch is operating in NPV mode.

NPV traffic maps can be configured only in NPV mode.

2. Something the configuration guide forgot to mention is that we need to shut down the vfc before configuring npv traffic-map. Otherwise, you will get an error.

3. how to verify?

Verifying NPV Traffic Management

To display the NPV traffic map, enter the show npv traffic-map command.

NPV Traffic Map Information:

----------------------------------------

Server-If       External-If(s)

----------------------------------------

fc1/3           fc1/10,fc1/11

fc1/5           fc1/1,fc1/2

----------------------------------------

To display the NPV internal traffic details, enter the show npv internal info traffic-map command.

NPV Traffic Map Information:

----------------------------------------

Server-If       External-If(s)

----------------------------------------

fc1/3           fc1/10,fc1/11

fc1/5           fc1/1,fc1/2

----------------------------------------

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/nx-os/configuration/guides/int/int_cli_4_2_published/cli_interfaces/npv.html

=====UCS=====

great write-up here:

f you use SAN pin groups in UCS manager. it wil translate to the NPV traffic map feature on the CLI. You can see that with the “show npv traffic-map” command on the FI (connect nxos)

show npv traffic-map

UCS-SB60-A(nxos)# show npv traffic-map

NPV Traffic Map Information:
—————————————-
Server-If External-If(s)

—————————————-
vfc699 san-port-channel 100
vfc700 vfc697
vfc701 vfc697
vfc702 san-port-channel 100
—————————————-

show running:

npv traffic-map server-interface vfc699 external-interface san-port-channel 100
npv traffic-map server-interface vfc700 external-interface vfc697
npv traffic-map server-interface vfc701 external-interface vfc697
npv traffic-map server-interface vfc702 external-interface san-port-channel 100

↧

FCoE raw note

August 16, 2014, 3:57 am

≫ Next: default gateway for Nexus 7000 management

≪ Previous: npv traffic-map

http://blog.scottlowe.org/2009/10/25/setting-up-fcoe-on-a-nexus-5000/

FCoE config is well explained here.

↧

default gateway for Nexus 7000 management

August 16, 2014, 4:23 am

≫ Next: UCS vNIC when to enable failover?

≪ Previous: FCoE raw note

There is always a " BUT".... :)

Just a quick note about how to configure the default gateway for N7K management..

Storage VDC

Configuring a VDC for the OOB management interface mgmt0 is accomplished with the vrf context management command. However, a storage VDC does not support VRF, so configuring mgmt0 requires a different approach.

The following table shows how to configure mgmt 0 for a VDC and for a storage VDC:

Configuring mgmt 0 for VDC

Configuring mgmt 0 for storage VDC

vrf context management
ip route 0.0.0.0/0 default_gateway

interface mgmt 0
ip address mgmt0_ip_addressmgmt0_subnet_mask
no shut
ip route 0.0.0.0/0 default_gateway

Note

The ip route command specifies the default route that points to the default gateway.

where

mgmt0_ip_address is the mgmt0 IPv4 address.
mgmt0_subnet_mask is the mgmt0 IPv4 netmask.
default_gateway is the IPv4 address of the default-gateway.

↧

UCS vNIC when to enable failover?

August 16, 2014, 4:49 am

≫ Next: nexus 7000 port group allocation && VDC

≪ Previous: default gateway for Nexus 7000 management

when creating a service profile in UCS, one of the options for vNIC is " enable failover", so when do you do it? and when not to?

according to cco:

The fabric interconnect associated with the component.

If you want vNICs created from this template to be able to access the second fabric interconnect if the default one is unavailable, check the Enable Failover check box.

adapter that supports fabric failover:

Palo adapter (M71KR and M81KR)

Note

Do not enable vNIC fabric failover under the following circumstances:

If the Cisco UCS domain is running in Ethernet Switch Mode. vNIC fabric failover is not supported in that mode. If all Ethernet uplinks on one fabric interconnect fail, the vNICs do not fail over to the other.
if you plan to associate one or more vNICs created from this template with a server that has an adapter which does not support fabric failover, such as the Cisco UCS 82598KR-CI 10-Gigabit Ethernet Adapter. If you do so,Cisco UCS Manager generates a configuration fault when you associate the service profile with the server.

Great post:
https://supportforums.cisco.com/document/72501/understanding-fabric-failure-and-failover-ucs

Understanding Fabric Failure

In a simple scenario of UCS system with a server with CNA card, following may happen:

a) FI failure : results in fabric failure for all connected UCS chassis

b) FEX failure : results in fabric failure for one UCS chassis

c) FI-FEX link failure : results in fabric failure for some of the servers within a UCS chassis (depending on number of servers and uplinks)

d) One CNA port failure : results in fabric failure for one server

In any of the above cases downtime can be eliminated by using redundant hardware and proper config.

Understanding Failover

When redundant hardware and proper configuration is in place, any failure will result in failover. The behaviour described below is for end-host mode only, since in switched mode the link status is not propagated.

a) One uplink of one FI fail : In this case UCS will re-pin the traffic to the remaining uplink to the FI.

b) Both uplinks of one FI fail or FI fails : In this case the corresponding server links will be shut since there is no uplink available on an FI. The FI will propagate link-down status to the adapter. Once adapter link-down status occurs, it is the responsibility of the operating system to re-pin traffic to the remaining NIC/HBA. The exception here is with Palo adapter (M71KR and M81KR) which supports fabric failover.

c) One uplink of one FEX fails : In this case the server blades pinned to the failed uplink will have the links shut. Although this applies only to UCS not having the new hardware FEX & FI, running 1.x or 2.x.

d) Both uplinks of one FEX fail or FEX fails : In this case all adapters on that fabric will lose network/storage connectivity. If host level redundancy is configured (NIC teaming and SAN multi-pathing) the traffic will be re-routed trough the other FEX.

e) One adapter fails : If this is the only adapter then connectivity will be lost. If a redundant adapter is available and host level redundancy is configured, the traffic will be re-routed through the other adapter. Some UCS adapters like M71KR and M81KR support fabric failover at adapter level, thus eliminating the need of host level redundancy configuration (NIC teaming). As in case of NIC teaming, this will detect any failure between the adapter and the FI uplink. However, SAN fabric design considerations must be considered for vHBA failover. In most situations it is discouraged to have vHBA fabric failover.

Related Information

How does UCS manager High Availability architecture works

UCS with a single fabric interconnect vs. dual fabric interconnect topology

↧

nexus 7000 port group allocation && VDC

August 21, 2014, 1:11 am

≫ Next: issue when configuring ISL trunk port between N5K and MDS

≪ Previous: UCS vNIC when to enable failover?

Q: How do I know which port group a specific port belongs to ?

A: cco said:

Table: Port Numbers for Cisco Nexus 7000 Series 32-port 10-Gbps Ethernet module

Port Group	Port Numbers
1	1, 3, 5, 7
2	2, 4, 6, 8
3	9, 11, 13, 15
4	10, 12, 14, 16
5	17, 19, 21, 23
6	18, 20, 22, 24
7	25, 27, 29, 31
8	26, 28, 30, 32

Q2: why I care?

A: when you allocate an interface to a VDC, it will allocate the whole port group.

eg:

===commands====

n7k-2(config)# vdc fp

n7k-2(config-vdc)# alloc int e4/10

===commands====

Entire port-group is not present in the command. Missing ports will be included automatically

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the ports (y/n)? [yes] yes

n7k-2(config-vdc)# sh vdc membership

vdc_id: 0 vdc_name: Unallocated interfaces:

====snip====

vdc_id: 2 vdc_name: fp interfaces:

Ethernet4/10 Ethernet4/12 Ethernet4/14

Ethernet4/16

====snip====

↧

issue when configuring ISL trunk port between N5K and MDS

August 21, 2014, 2:53 am

≫ Next: FCoE NPIV NPV port channel between n5k and n7k

≪ Previous: nexus 7000 port group allocation && VDC

This was driving me mad in the lab!

the task is simple, to configure a FC port channel between a n5k and a MDS. but no matter what I do, the fc ports will stay offline, or stuck at initialising.

My FC ISL trunk port channel just won't come up. :(

In the end, after 100% sure about my configuration, I decided to save the config and reload the module in N5K (the fc ports are part of the unified port module) as suggested by my friend, then, it is working...

well, the order of operation is also mighty important, if you follow my tested step below, plus a reload of the n5k module, you should be fine.
==rough note: sample config ===

===mds ===

vsan database
vsan ##
int po#
channel mode act
sw tru mod on
sw tr all vsan ##
sw tr all vsan add##
sw rate-mode dedicated ----------- > kindly thanks to Mitch to spot this missing-linesw mode E

no sh

int fc ##

channel-group # force
no sh

==n5k===

**if you are using unified port module, mine is in slot 2 **

feature fcoe

slot 2
port ## type fc
pow mod 2
----wait a bit!------
no pow mod 2

int san-po##
chan mode ac
sw mode e
sw tr mode on
sw tr all vsan ##
sw tr all vsan add ##
no sh

int fc##
channel-group ## force

====after this, you may see the dreaded "offline" or " intializing"for the fc ports,even after you flap the ports like crazy:

n5k-1(config-vsan-db)# sh int fc2/5
fc2/5 is down (Offline)
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is ****
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port vsan is 1
Receive data field Size is 2112
Beacon is turned off
Belongs to san-port-channel 10
1 minute input rate 48 bits/sec, 6 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
3 frames input, 396 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
0 frames output, 0 bytes
0 discards, 0 errors
0 input OLS, 0 LRR, 0 NOS, 0 loop inits
7 output OLS, 7 LRR, 0 NOS, 0 loop inits
last clearing of "show interface" counters never

mds-1(config-if)# sh int fc1/1
fc1/1 is down (Initializing)
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:01:54:7f:ee:70:24:80
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port vsan is 1
Receive data field Size is 2112
Beacon is turned off
Belongs to port-channel 10
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 8 bits/sec, 1 bytes/sec, 0 frames/sec
1 frames input, 140 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
5 frames output, 652 bytes
0 discards, 0 errors
17 input OLS, 9 LRR, 2 NOS, 0 loop inits
9 output OLS, 0 LRR, 9 NOS, 0 loop inits

=====now it is time to save the config on n5k, and reload the module =====

copy run start
n5k-1(config)# pow mod 2

----------wait a bit-----------------------

n5k-1(config)# no pow mod
----------wait a bit-----------------------
n5k-1(config)#
n5k-1(config)#
n5k-1(config)#
n5k-1(config)#
n5k-1(config)# sh int fc2/5
fc2/5 is trunking
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is ***
Peer port WWN is *****
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Speed is 4 Gbps
Transmit B2B Credit is 250
Receive B2B Credit is 16
Receive data field Size is 2112
Beacon is turned off
Belongs to san-port-channel 10
Trunk vsans (admin allowed and active) (1,100)
Trunk vsans (up) (1,100)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 5608 bits/sec, 701 bytes/sec, 8 frames/sec
1 minute output rate 6080 bits/sec, 760 bytes/sec, 8 frames/sec
523 frames input, 42104 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
524 frames output, 45604 bytes
0 discards, 0 errors
0 input OLS, 1 LRR, 0 NOS, 0 loop inits
1 output OLS, 1 LRR, 0 NOS, 0 loop inits
last clearing of "show interface" counters never
16 receive B2B credit remaining
250 transmit B2B credit remaining
0 low priority transmit B2B credit remaining

Happy trunking!!

↧

FCoE NPIV NPV port channel between n5k and n7k

August 21, 2014, 3:55 am

≫ Next: 101st post: pass the CCIE DC exam yesterday : (updated)

≪ Previous: issue when configuring ISL trunk port between N5K and MDS

sample config for FCoE NPIV NPV port channel between n5k and n7k , you are going to have a hard time troubleshooting why your vsan is stuck at initializing if you miss some of the commands~

=====n7k====

feature lldp -----> easy to forget this since in n5k it is enabled by default ~~

feature npiv

feature lacp

vlan 10
fcoe vsan 10

vsan database
vsan 10
int E1/1-2
channel-group 1 mode active
no shut

int po1
switch mode tr
sw tr all vlan 10

int vfc-po1 ---> with this , we can be lazy and not configure the binding, since it will be applied automatically

sw mode f
sw tru mode on
sw tr all vsan 100

no shut

===N5K===

feature npv---->require a reload here

int e1/1-2
chann 1 mode ac
no sh

vlan 10
fcoe vsan 10

vsan database
vsan 10

int po 1
sw mode trunk
sw tru all vlan 10

int vfc1
bind int po1
sw mode np
sw tr all vsan 100
no sh

↧

101st post: pass the CCIE DC exam yesterday : (updated)

August 25, 2014, 6:37 pm

≫ Next: UCS adapter / link collection

≪ Previous: FCoE NPIV NPV port channel between n5k and n7k

It is quite a coincident that I am able to announce in my 101st post that I have passed the CCIE DC exam yesterday in Sydney. :)

being aware of the NDA, I just want to share some of my general experience preparing for the exam and the feeling exam itself.

first of all, so much has been said and so true in those who has conquered this before me. They are truly inspiring and got most covered already:
INE Brian's gold post
http://vmtrooper.com/my-ccie-data-center-lab-exam-experience/
http://jeremywaldrop.wordpress.com/2014/03/17/my-journey-to-ccie-data-center/
http://rickmur.wordpress.com/2013/05/17/passed-ccie-data-center/

My 2 cents:

really understand the concepts, configuration may be simple, but you need to know what to do QUICK and EFFECTIVE in the exam when the configuration seems correct but it just doesn't work.
Based on my experience with the devices, just be careful when you reload stuff, things may get lost, double check after reload to verify.
verify each step because a lot of things are inter-dependent! don't wait till it is too late to do so.
when practice, time yourself and get to the optimal speed of getting things done CORRECTLY and also efficiently. you don't have much time, but you also can't afford to lose time correcting your own misconfiguration.
practice doing everything in one single 24 inch monitor. That was quite a challenge for those are spoilt with 2 or 3 monitors daily.
DON'T LINGER. since it is in the lab, something is not meant to work. SO resist the urge to TROUBLESHOOT and just get things done.
Proctor is there to help. so if you get stuck for more than 10 minutes and suspect it is hardware or a bug, ASK. you won't get into trouble.
SAVE config OFTEN. during my lab, one device did crash. and the first thing I did in each device is to follow my friend's advice: create a cli alias for fast access:

cli alias name cc copy run start

another alias i made is to help me get into a VDC quickly:

cli alias name san switchto vdc storage

do everything in a systematical way so you don't forget things. switch 1 first, then switch 2, layer 2 first, then layer 3. whatever works for you and keep you in track.
if you get panic, take mini break or a few deep breaths, and STOP,because that was the perfect mood to make mistakes, which can be costly to correct.
use " show log last ##" if things break. and compare configuration on both ends if you are not sure.
try to get a good night sleep, it helps, but not a big deal really, I couldn't sleep much the night before my exam and even if I did fall alseep eventually around 3 AM, I was dreaming about typing commands (totally intoxicated ) , but I still manage to pass, so I guess the most important thing is to RELAX and have FAITH in yourself.
many people suggested to draw down the topology before starting, I personally didn't do that. it just takes too much time and not very easy to ready in a A4 piece of paper any way. I am used to read stuff from a shiny screen now and the topology in the lab is very easy to read.
Speed!! even if you know something well, you still need to be practice the configuration again and again and aim to get it right every time.
be care of the "copy and paste" from notepad to CLI, I used to like that in my daily job, because i am lazy, but in rare cases, for some devices, be cautious, i found the device can eat one or two line of the config I pasted in, and I ended up having to troubleshoot it to find that missing line, more time wasted, or I paste config from switch A to its peer, but forgot to edit the tiny bit of change, such as IP, then, again, trip myself over on that. with the " tab" to auto complete, you don't really save much time with "copy and paste".
one funny tip I heard from my peer is to practice doing your lab in an old traditional keyboard, it is something good to keep in mind.
MAC vs Windows. if you are a mac person, try to practice in your VM. there is no time in the lab for you to try to get used to Windows again.
I have to stress keeping your health to the optimal is very essential part of the preparation.
take some snacks with you INTO the lab, since you can really get hungry quickly after getting up super early to get to your exam and the lunch is after 12 pm and tiny.

feel free to comment for your questions (be mindful of the NDA again).

all the best to your preparation and the exam.

Ming Tang
CCIE DC # 44562

↧

UCS adapter / link collection

August 26, 2014, 5:30 pm

≫ Next: end to end FCoE troubleshooting check with UCS, Nexus 5000 etc etc

≪ Previous: 101st post: pass the CCIE DC exam yesterday : (updated)

excellent link to better understand different type of UCS adapters available:

↧

end to end FCoE troubleshooting check with UCS, Nexus 5000 etc etc

August 27, 2014, 6:25 pm

≫ Next: raw note: VTP on Nexus 5000 switches

≪ Previous: UCS adapter / link collection

UCS:
* FCoE port channel created? assigned the right vsan?
* fcoe vlan created and mapped to vsan?

N5k/MDS/upstream FCoE switch:

* Ethernet port up? in port channel? allow the fcoe vlan?
* vlan created?
*vsan created in vsan database?
* VFC up? vsan trunking? ----> show interface vfc #
* see the flogi of the servers and storage? ---> " show flogi database"
* zoning configured with the right zone/zoneset/vsan? ---> show zone
* zoneset activated? ---> " show zoneset activate", you should be able to see a * next to the logged in pwwn
note: may need to reboot the server after activating the zoneset
* zoneset distributed? ---> if you are running basic zoning

storage: (EMC/Netapp etc )
* should be able to see the initiators if everything has been configured properly in servers and switch

↧

raw note: VTP on Nexus 5000 switches

August 28, 2014, 9:17 pm

≫ Next: inter-vlan routing failure on n5k/l3 switch

≪ Previous: end to end FCoE troubleshooting check with UCS, Nexus 5000 etc etc

* config guide in 4.x

very brief

This example shows how to configure VTP in transparent mode (the default mode ):

switch#config t
switch(config)#feature vtp
switch(config)#vtp domain accounting
switch(config)#vtp version2
switch(config)#exit
switch#

** this behavior  will possibly lead to the following issue:

Nexus 5000 does not have the same VLANs as switch running VTP server

VLANs for the Nexus 5000 are not the same as for the switch running the VTP server.

Possible Cause

The Nexus 5000 currently supports VTP only in transparent mode (4.2(1)N1(1) and later releases).

Solution

This situation indicates that VLANs must be configured locally. However a VTP client and server can both communicate through a Nexus 5000 by using the following commands:

```
  better config guide : 
```

About the VLAN Trunking Protocol

VTP is a distributed VLAN database management protocol that synchronizes the VTP VLAN database across domains. A VTP domain includes one or more network switches that share the same VTP domain name and that are connected with trunk interfaces. Each switch can be in only one VTP domain. Layer 2 trunk interfaces, Layer 2 port channels, and virtual port channels (vPCs) support VTP functionality. Cisco NX-OS Release 5.0(2)N1(1) introduces the support for VTPv1 and VTP2. Beginning in Cisco NX-OS Release 5.0(2)N2(1), you can configure VTP in client or server mode. Prior to NX-OS Release 5.0(2)N2(1), VTP worked only in transparent mode.

There are four VTP modes:

Server mode–Allows users to perform configurations, it manages the VLAN database version #, and stores the VLAN database.
Client mode–Does not allow user configurations and relies on other switches in the domain to provide configuration information.
Off mode—Allows you to access the VLAN database (VTP is enabled) but not participate in VTP.
Transparent mode–Does not participate in VTP, uses local configuration, and relays VTP packets to other forward ports. VLAN changes affect only the local switch. A VTP transparent network switch does not advertise its VLAN configuration and does not synchronize its VLAN configuration based on received advertisements.

Guidelines and Limitations

Guidelines and Limitations

VTP has the following configuration guidelines and limitations:

When a switch is configured as a VTP client, you cannot create VLANs on the switch in the range of 1 to 1005.
VLAN 1 is required on all trunk ports used for switch interconnects if VTP is supported in the network. Disabling VLAN 1 from any of these ports prevents VTP from functioning properly.
If you enable VTP, you must configure either version 1 or version 2. On the Cisco Nexus 5010 and Nexus 5020 switch, 512 VLANs are supported. If these switches are in a distribution network with other switches, the limit remains the same.
On the Cisco Nexus 5010 switch and the Nexus 5020 switch, 512 VLANs are supported. If these switches are in a distribution network with other switches, the VLAN limit for the VTP domain is 512. If a Nexus 5010 switch or Nexus 5020 switch client/server receives additional VLANs from a VTP server, they transition to transparent mode
The show running-configuration command does not show VLAN or VTP configuration information for VLANs 1 to 1000.
When deployed with vPC, both vPC switches must be configured identically.
VTP advertisements are not sent out on Cisco Nexus 2000 Series Fabric Extender ports.
VTP pruning is not supported.

 ** interesting discussion about a bug:

There is a bug on the N3K that causes this behavior even if they're currently set the same if you have ever enabled VTP the box still thinks its on. Perhaps that bug also exists on N5K if show vpc status shows the same state.

In any case since its a type 2 inconsistency it doesn't affect traffic flow.

 *real life case of how VTP can cause an outage with some human error:

↧

inter-vlan routing failure on n5k/l3 switch

August 28, 2014, 9:38 pm

≫ Next: vSphere HA/DRS cluster status color:

≪ Previous: raw note: VTP on Nexus 5000 switches

issue:
the host can ping the ip in the same vlan,but can't ping other hosts in other vlans.
and N5k is the core and has the SVIs as default gateways.

troubleshooting steps:

1. check if default gateway is configured properly in the host and N5K.
2. check log to see if there is any duplicate IP ARP, any other host is using the default gateway's ip address?

↧