I was recently asked by a client to help troubleshoot an issue they were having after upgrading the IOS on two of their core Cisco Catalyst 6509s. The two switches are connected together by three 1Gb fibre links that run about 500m between two buildings (Building A and B). Two of the links are bundled together in an 802.1q Etherchannel. Only one vlan, 100, is allowed to cross the link and this vlan is used purely for traffic to and from a cluster of Call Manager 4 servers. The other link is a 1Gb layer 3 fibre link that is used for any other user traffic across the core switches.

Network Layout

The 6509s also have links to other core switches within the network, so if the layer 3 link goes down traffic should be routed through these other switches.

After the IOS upgrades took place users at Building B had reported that access to a server at Building A was slow or dropping out. The Layer 3 link was up with no errros reported and to the client everything looked fine.

After looking at the routing between both locations I discovered that the traffic was crossing the layer 2 etherchannel passing through the vlan 100 interfaces of both switches. The way the switches were configured, this route was the optimal path for traffic to flow through, although my client was adamant that this had not been the case before. To OSPF, the interior gateway protocol that was used, this path had the lowest cost (due to the higher available bandwidth), and so this path was chosen.

Building A Switch

interface Vlan100
description ** IPTVoice_Server_Vlan **
ip address 10.100.10.3 255.255.255.192
no ip redirects
ip route-cache flow
standby 99 ip 10.100.10.1
standby 99 preempt

router ospf 1
!
network 10.100.10.0 0.0.0.63 area 17

Building B Switch

interface Vlan100
description ** IPT/IPCC Voice_Server_Vlan1 **
ip address 10.100.10.2 255.255.255.192
no ip redirects
standby 99 ip 10.100.10.1
standby 99 priority 105
standby 99 preempt

router ospf 1
!
network 10.100.10.0 0.0.0.63 area 17

Another closer look showed that one of the interfaces of the Etherchannel on the Building A switch was showing a high amount of errors.  Replacing the fibre patch cable soon sorted the issue and users started reporting that all was well again with their connection to the server.

This, however, did not solve the mystery of user traffic suddenly passing over the Voice Server Vlan. A quick download of some archived configurations from Ciscoworks soon revealed what had happend. Before the IOS upgarde interface vlan 100 on Building A 6509 was shutdown. After some confused looks and some discussion, the only reason we could come up with for this being done was that when the voice server solution was deployed by an outside vendor, the vendor encountered the same problem of user traffic flowing over the etherchannel. The design documents stated that the link should only be used for voice server traffic and that HSRP should be up and running. The deployment engineers may have brought the link up and noticed user traffic flowing over it. Instead of working round the problem they may have found the easiest solution was to shut down one of the interface vlan 100 interfaces. I have left the matter of why the deployment was like this with my not so happy client to dicuss with the vendor.

Adjusting the ospf costs would not have made a difference in this case. The users and the users’ servers were in ospf area 17, while the layer 3 link was a backbone link in area 0. Intra-area links will always be preferred over Inter-area ones. We could have, of  course, put the servers network in a different area but, the fastest and simplest solution I could come up with for this problem involved the ospf passive command. OSPF should only generally be run on transit links and I’ve used the passive command in many environments where layer 3 switches have lots of vlan intefaces that have formed braodaast network type adjancies. This also helps to cut down on CPU and memory usage.

The command is implement under the router ospf configuration mode and it’s an easy one to understand. To stop the two vlan 100 interfaces forming an adjaceny I only had to add it to one side of the link.

Building A Switch

router ospf 1
!
passive-interface Vlan100

That one command was enough to take down the ospf adjacency between the two Vlan 100 interfaces and stop the user traffic flowing across the link. The 10.100.100/24 subnet is still advertised out to the rest of the network and now HSRP is running as the designer planned, as both interfaces as are up.