One of my colleagues was able to access two of the core data centre switches but I could only get to one. A very quick trip to the data centre floor and a console cable connection into the core data centre switches revealed the issue.

The two switches in question (Catalyst 4507s) have a 2 x 1Gb EtherChannel configured on either end, connected by fibre connections. One side of the connection reported that both links were active in the Etherchannel. The other side had one link as down and the logs showing that the connection had left the EtherChannel.

The full reason for this is still unknown but this type of issue, where one side sees the link as up but the other sees it as down, is called a unidrectional failure. To solve the matter at hand, we first of all shut down the faulty link at the end that still had the link up. As soon as this was done everything sprung into life and everyone was able to connect to the data centre hosts. While the link was down we quickly swapped out the GBic cards and brought the connection backup. The link joined back into the EtherChannel and everything was back as it should be.

This highlighted an issue with the Etherchannel configuration on these particular switches however. Here is a look at the configuration of one of the Etherchannel interfaces as it stood at that time.

interface GigabitEthernet1/1
description ** link to xxxxxx **
switchport trunk encapsulation dot1q
switchport trunk allowed vlan x,xx,xx,xx-xx,xx-xx,xx
switchport mode trunk
qos trust dscp
channel-group 1 mode on

I have always advised network engineers to use a mode of desirable on either side of an  Etherchannel connection, rather than forcing the Etherchannel up.  The on mode forces a port to join an Etherchannel without any sort of Etherchannel protocol negotiation taking place. Using the desirable keyword instead of the on keyword means that the switch uses the Port Aggregation Protocol (PAgP). When using PAgP the switch learns of partner interfaces on other switches that support PAgP and dynamically groups its interfaces into an Etherchannel. Lets look at an example. I’ve set up two Cisco Catalyst 3550s back to back connecting ports 13 and 14 off each switch together.

Here is the configuration of the ports on either end of the connection.

interface FastEthernet0/13
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode desirable
!
interface FastEthernet0/14
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode desirable

Once the port channel group on the first configured interface the IOS automatically creates the port channel interafce.

interface Port-channel1
switchport trunk encapsulation dot1q
switchport mode trunk

If the same configuration is applied at both ends then the PAgP protocol will dynamically place each relevant interface into the Etherchannel.

Here is the output of the show etherchannel summary command from SW1

SW1#show etherchannel summary
Flags:  D – down        P – in port-channel
I – stand-alone s – suspended
H – Hot-standby (LACP only)
R – Layer3      S – Layer2
U – in use      f – failed to allocate aggregator
u – unsuitable for bundling
w – waiting to be aggregated
d – default port
Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
——+————-+———–+———————————————–
1      Po1(SU)         PAgP      Fa0/13(P)   Fa0/14(P)

Lets see what happens if I change port fas0/14 on SW2, removing it from the channel and thus stopping PAgP.

SW2(config-if)#no channel-group 1
SW2(config-if)#do show etherchannel summ
Flags:  D – down        P – in port-channel
I – stand-alone s – suspended
H – Hot-standby (LACP only)
R – Layer3      S – Layer2
U – in use      f – failed to allocate aggregator
u – unsuitable for bundling
w – waiting to be aggregated
d – default port
Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
——+————-+———–+———————————————–
1      Po1(SU)         PAgP      Fa0/13(P)

On SW1 both fas0/13 and 14 are still configured as part of port channel 1. But as PAgP is used, SW1 drops port14 from the group when it stops seeing PAgP.

SW1#sh etherchannel summ
Flags:  D – down        P – in port-channel
I – stand-alone s – suspended
H – Hot-standby (LACP only)
R – Layer3      S – Layer2
U – in use      f – failed to allocate aggregator
u – unsuitable for bundling
w – waiting to be aggregated
d – default port
Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
——+————-+———–+———————————————–
1      Po1(SU)         PAgP      Fa0/13(P)   Fa0/14(I)

Port fas0/14 is now operating as a stand-alone port and is now a seperate trunk between the switches.

SW1#sh int trunk

Port        Mode         Encapsulation  Status        Native vlan
Fa0/14      on           802.1q         trunking      1
Po1         on           802.1q         trunking      1

SW2#sh int trunk

Port        Mode         Encapsulation  Status        Native vlan
Fa0/14      on           802.1q         trunking      1
Po1         on           802.1q         trunking      1

In the data centre situation I described above this would have dropped the offending interfaces from the etherchannel as one side would have stopped seeing PAgP. However, it may have been possible for one switch to move the interface into stand-alone mode and pass traffic across a broken link, as it was still seeing this link as up. In order to help in situations like these Cisco developed the Unidirectional Link Dection protocol.

UDLD can now be configured in aggressive mode from IOS Release 12.1(3a)E.  Cisco describe aggressive mode as follows:

” Configure UDLD aggressive mode only on point-to-point links between network devices that support UDLD aggressive mode. With UDLD aggressive mode enabled, when a port on a bidirectional link that has a UDLD neighbor relationship established stops receiving UDLD packets, UDLD tries to reestablish the connection with the neighbor. After eight failed retries, the port is disabled.

To prevent spanning tree loops, nonaggressive UDLD with the default interval of 15 seconds is fast enough to shut down a unidirectional link before a blocking port transitions to the forwarding state (with default spanning tree parameters).

When you enable UDLD aggressive mode, you receive additional benefits in the following situations:

•One side of a link has a port stuck (both Tx and Rx)

•One side of a link remains up while the other side of the link has gone down

In these cases, UDLD aggressive mode disables one of the ports on the link, which prevents traffic from being discarding. “

I use aggressive mode when available. Configuration is simple. To configure non-aggressive udld you would enter the following command.

SW2(config)#int fas0/13
SW2(config-if)#udld port

To configure aggressive mode only one more keyword is required

SW2(config-if)#int fas0/14
SW2(config-if)#udld port aggressive

Using the show udld command you can check to make sure that udld is running as desired.

SW2#sh udld

Interface Fa0/13
—
Port enable administrative configuration setting: Enabled / in aggressive mode
Port enable operational state: Enabled / in aggressive mode
Current bidirectional state: Bidirectional
Current operational state: Advertisement – Single neighbor detected
Message interval: 7
Time out interval: 5

Entry 1
—
Expiration time: 44
Cache Device index: 1
Current neighbor state: Bidirectional
Device ID: CAT0640X09Y
Port ID: Fa0/13
Neighbor echo 1 device: CAT0825X28N
Neighbor echo 1 port: Fa0/13

Message interval: 15
Time out interval: 5
CDP Device name: SW1

Interface Fa0/14
—
Port enable administrative configuration setting: Enabled / in aggressive mode
Port enable operational state: Enabled / in aggressive mode
Current bidirectional state: Bidirectional
Current operational state: Advertisement – Single neighbor detected
Message interval: 15
Time out interval: 5

Entry 1
—
Expiration time: 33
Cache Device index: 1
Current neighbor state: Bidirectional
Device ID: CAT0640X09Y
Port ID: Fa0/14
Neighbor echo 1 device: CAT0825X28N
Neighbor echo 1 port: Fa0/14

Message interval: 15
No timeout interval
No CDP device name

The final configuration of the ports on either end looks like this:

interface FastEthernet0/13
switchport trunk encapsulation dot1q
switchport mode trunk
udld port aggressive
channel-group 1 mode desirable
!
interface FastEthernet0/14
switchport trunk encapsulation dot1q
switchport mode trunk
udld port aggressive
channel-group 1 mode desirable

Etherchannels are wonderful things and in the most part run without any hitches. However, I think that running some sort of protocol to help dynamically manage the participating interfaces and using UDLD to monitor for unidirectional failures is a good safeguard from situations such as the one described above.

I was recently asked by a client to help troubleshoot an issue they were having after upgrading the IOS on two of their core Cisco Catalyst 6509s. The two switches are connected together by three 1Gb fibre links that run about 500m between two buildings (Building A and B). Two of the links are bundled together in an 802.1q Etherchannel. Only one vlan, 100, is allowed to cross the link and this vlan is used purely for traffic to and from a cluster of Call Manager 4 servers. The other link is a 1Gb layer 3 fibre link that is used for any other user traffic across the core switches.

Network Layout

The 6509s also have links to other core switches within the network, so if the layer 3 link goes down traffic should be routed through these other switches.

After the IOS upgrades took place users at Building B had reported that access to a server at Building A was slow or dropping out. The Layer 3 link was up with no errros reported and to the client everything looked fine.

After looking at the routing between both locations I discovered that the traffic was crossing the layer 2 etherchannel passing through the vlan 100 interfaces of both switches. The way the switches were configured, this route was the optimal path for traffic to flow through, although my client was adamant that this had not been the case before. To OSPF, the interior gateway protocol that was used, this path had the lowest cost (due to the higher available bandwidth), and so this path was chosen.

Building A Switch

interface Vlan100
description ** IPTVoice_Server_Vlan **
ip address 10.100.10.3 255.255.255.192
no ip redirects
ip route-cache flow
standby 99 ip 10.100.10.1
standby 99 preempt

router ospf 1
!
network 10.100.10.0 0.0.0.63 area 17

Building B Switch

interface Vlan100
description ** IPT/IPCC Voice_Server_Vlan1 **
ip address 10.100.10.2 255.255.255.192
no ip redirects
standby 99 ip 10.100.10.1
standby 99 priority 105
standby 99 preempt

router ospf 1
!
network 10.100.10.0 0.0.0.63 area 17

Another closer look showed that one of the interfaces of the Etherchannel on the Building A switch was showing a high amount of errors.  Replacing the fibre patch cable soon sorted the issue and users started reporting that all was well again with their connection to the server.

This, however, did not solve the mystery of user traffic suddenly passing over the Voice Server Vlan. A quick download of some archived configurations from Ciscoworks soon revealed what had happend. Before the IOS upgarde interface vlan 100 on Building A 6509 was shutdown. After some confused looks and some discussion, the only reason we could come up with for this being done was that when the voice server solution was deployed by an outside vendor, the vendor encountered the same problem of user traffic flowing over the etherchannel. The design documents stated that the link should only be used for voice server traffic and that HSRP should be up and running. The deployment engineers may have brought the link up and noticed user traffic flowing over it. Instead of working round the problem they may have found the easiest solution was to shut down one of the interface vlan 100 interfaces. I have left the matter of why the deployment was like this with my not so happy client to dicuss with the vendor.

Adjusting the ospf costs would not have made a difference in this case. The users and the users’ servers were in ospf area 17, while the layer 3 link was a backbone link in area 0. Intra-area links will always be preferred over Inter-area ones. We could have, of  course, put the servers network in a different area but, the fastest and simplest solution I could come up with for this problem involved the ospf passive command. OSPF should only generally be run on transit links and I’ve used the passive command in many environments where layer 3 switches have lots of vlan intefaces that have formed braodaast network type adjancies. This also helps to cut down on CPU and memory usage.

The command is implement under the router ospf configuration mode and it’s an easy one to understand. To stop the two vlan 100 interfaces forming an adjaceny I only had to add it to one side of the link.

Building A Switch

router ospf 1
!
passive-interface Vlan100

That one command was enough to take down the ospf adjacency between the two Vlan 100 interfaces and stop the user traffic flowing across the link. The 10.100.100/24 subnet is still advertised out to the rest of the network and now HSRP is running as the designer planned, as both interfaces as are up.