6024 Routing stopped...why?

Okay, the rest of this message. These laptop mouse pads drive me crazy..brushed it and it sent the message for me...go figure ..anyway as I was saying...

Setup as follows
VLAN10 192.168.50.0 /24 if IP 192.168.50.1
VLAN20 192.168.60.0 /24 if IP 192.168.60.1
g15 192.168.200.0 /24 if IP 192.168.200.1 DHCP server 192.168.200.5 connects to g15 directly.
g23 192.168.10.0 /24 if IP 192.168.10.1 WAN GW IP 192.168.10.2

DHCP relay enabled.
Hosts can pick up the appropriate IP from the appropriate scope according to what network they are on.
Hosts can ping their local interface and other hosts on their subnet, but not any other hosts on another networks. When I try pinging, I get a "Dest. net unreachable reply" which indicates to me the route does not exist and so no routing is happening. Where did the route go and why isn't it being created by the router?

When I first created the VLANs, I could ping, map network drives etc between VLANs, however, it seemed to *spontaneously* stop working. For example it was working fine (ie routing between networks) when I left the office, and then the next day, when I came in, it had stopped.

After I tried pinging a few times, it suddenly started working again. I haven't done anything with ACLs or anything like that and it hasn't started working again since.

Should OSPF be enabled? Do I have to do router voodoo configuration?

Any help appreciated.

TimeTraveller

Can you post your entire config for the 6024?  Also can you tell me what is connected to the 6024?  I see you have a DHCP server on one port and the internet (what is this device is it a router or the ISP switch or a firewall or what) on another port.  Are the rest of the 6024 ports trunking to other switches then to your workstations or are the workstations all connected directly to the 6024?  Are there more then one router in the network (is the 6024 connected to another 6024 or another router)?  From your description perhaps all the other PCs are directly connected to the 6024?

Can you describe the symptoms more precisely?  It sounds like you had everything working and you could communicate between all the VLANs then after some idle period (assuming that nothing change anywhere in the network during that period) you lost connection between something - how did you determine you lost connection - did some services failed?  Did you lose ping?  How did you realize that something was wrong when you first noticed it.  After initial observed failure you said that when you ping some systems, they came back online correct?  You then mentioned that since then you lost connection again?  What happened during that time?  Anything change?  How did you notice the problem when you first observed it - what happen initially to tell you a problem occured?

So at this point what is the condition of the network?  Can you ping devices within the same VLAN?  Do all systems have a valid IP address - did they properly retrieve a valid IP from the DHCP server?  Can they ping each other across VLANs now or are you still unable to go between VLANs?  Are all the IP addresses unique on all the PCs - no duplicate IP?

Also can you tell me how each PC and the DHCP server are configured?  Did you configure a gateway for these devices?  Is the gateway the correct IP address for the 6024 interface to which they are connected (should be the IP address for the 6024 on their respective VLANs)?  Do you have any problem getting out to the Internet from these various systems before or now?

Cuong.

Cuong, many thanks for your reply. So many questions, I'll answer what I can. First, here is the config
Router Configuration
-----------------------------

interface ethernet g23
spanning-tree cost 200000
exit
interface range ethernet g(1-4)
switchport mode general
exit
vlan database
vlan 10,20
exit
interface ethernet g1
switchport general pvid 10
exit
interface ethernet g2
switchport general pvid 10
exit
interface ethernet g3
switchport general pvid 20

exit
interface ethernet g4
switchport general pvid 20
exit
interface range ethernet g(1-2)
switchport general allowed vlan add 10 untagged
exit
interface ethernet g3
switchport general allowed vlan add 20 untagged
exit
interface ethernet g4
switchport general allowed vlan add 20
exit
interface ethernet g15
ip address 192.168.200.1 255.255.255.0
exit
interface ethernet g23
ip address 192.168.10.1 255.255.255.0
exit
interface vlan 10
ip address 192.168.50.1 255.255.255.0
exit

interface vlan 20
ip address 192.168.60.1 255.255.255.0
exit
router ospf area 192.168.50.1
router ospf router-id 192.168.50.1
interface ip 192.168.10.1
ospf 192.168.50.1
exit
interface ip 192.168.50.1
ospf 192.168.50.1
exit
interface ip 192.168.60.1
ospf 192.168.50.1
exit
interface ip 192.168.200.1
ospf 192.168.50.1
exit
interface ip 192.168.10.1
exit
interface ip 192.168.50.1
exit
interface ip 192.168.60.1

exit
interface ip 192.168.200.1
exit
ip dhcp relay address 192.168.200.5
ip dhcp relay enable
username XXXXXXX password xyyyyyyyyy level 15 encrypted
snmp-server community Dell_Network_Manager rw view DefaultSuper
snmp-server set ospfAreaTable ospfAreaId 192.168.50.1 ospfImportAsExtern imp
ortExternal


OOB host Configuration
-----------------------------

interface out-of-band-eth 1
ip address 192.168.0.250 255.255.255.0
exit
interface out-of-band-eth 1
ip default-gateway 192.168.0.1
exit

_____________________________
>> Default settings:
_____________________________

>> Router Configuration
-----------------------------
>> Service tag: 639V291

>> SW version 2.0.0.01 (date 10-Apr-2005 time 08:28:21)

>> Gigabit Ethernet Ports
=============================
>> no shutdown
>> speed 1000
>> duplex full
>> negotiation
>> flow-control off

>> mdix auto
>> no back-pressure

>> interface vlan 1
>> interface port-channel 1 - 7

>> no router RIP

>> no router OSPF enable

>> spanning-tree
>> spanning-tree mode STP

>> qos basic

>> OOB host Configuration
-------------------------

>> interface out-of-band-eth
>> no shutdown
>> speed 100
>> duplex full
>> negotiation
>> flow-control off
>> no back-pressure
>> exit
console#



CN: Also can you tell me what is connected to the 6024?
TimeTraveller: Nothing much to start, am justing setting up with test for 1st use.
VLAN10 -- 1 PC directly connected 192.168.50.0 /24
VLAN20 -- 1 PC directtly connected 192.168.60.0 /24
G15 -- 1 DHCP Server, directly connected. 192.168.200.0p /24
Windows 2003 Server doing DHCP (successfully ) for entire network ( that's my plan anyway ).
G23 -- 1 D-Link 624 Wireless router direct connecte 192.168.10.0 /24 acting as WAN gateway, although in production this will be a SonicWall TZ170.

CN: Are the rest of the 6024 ports trunking to other switches then to your workstations or are the workstations all connected directly to the 6024?
TimeTraveller: At this point all hosts connect directly to 6024. In production we will trunk to a 3324 and have VLANs spanning using trunks.

CN: Are there more then one router in the network (is the 6024 connected to another 6024 or another router)? From your description perhaps all the other PCs are directly connected to the 6024?
TimeTraveller: Just the D-Link 624 on g23 if you call that a router. We are directly connected to it's LAN if, the WAN if is connected to nothing at this time.

CN:Can you describe the symptoms more precisely? It sounds like you had everything working and you could communicate between all the VLANs then after some idle period (assuming that nothing change anywhere in the network during that period) you lost connection between something - how did you determine you lost connection - did some services failed? Did you lose ping?
TimeTraveller: I noticed that I lost ping, that's the major symptom. Also, initially, I had mapped a network drive from a host on VLAN10 from VLAN20. In the morning, the network drive connection was lost, and I could not ping between hosts on different VLANs. After about 5 minutes of trying stuff, mainly pinging to see what would talk to what, it all magically started working again.

Later that day, it did stop working. I did some changes, (enabled OSPF) but can't remember if I did them before or after the thing stopped working.

In any case, I disabled OSPF,and regained the ability to ping individual interfaces on the 6024, but not the directly connnected hosts.

CN: So at this point what is the condition of the network? Can you ping devices within the same VLAN?
TimeTraveller: Yes.

CN: Do all systems have a valid IP address - did they properly retrieve a valid IP from the DHCP server?
TimeTraveller: Yes, appropriate to the network they lie on..eg hosts on 192.168.50.0 network receive a 192.168.50.xxx ip, those on 192.168.60.0 get a 192.168.60.xxx IP.

CN: Can they ping each other across VLANs now or are you still unable to go between VLANs?
TimeTraveller: Can't ping across VLAN's or from VLAN's to directly attached hosts.

CN: Are all the IP addresses unique on all the PCs - no duplicate IP?
TimeTraveller: All IP addresses are unique, there are no dupes.

CN: Also can you tell me how each PC and the DHCP server are configured? Did you configure a gateway for these devices?
TimeTraveller: The DHCP server has a superscope which dishes out IP address, SM, DNS, and gateway IP's to the appropriate hosts from the appropriate subscope on the appropriate subnet. The gateway that I am using is the interface of the VLAN or Port. eg VLAN10 IP if is 192.168.50.1 The gateway the host on VLAN10 gets from the DHCP server then is 192.168.50.1

CN:Do you have any problem getting out to the Internet from these various systems before or now?
TimeTraveller: I haven't actually gotten that far yet. I'm using the D-Link DI-624 as my "internet". I cannot connect to or ping it, although the DHCP server mentioned previously does send an IP to my laptop which is wirelessly connected to the DI-624. However, Icannot ping the DHCP server from my laptop, or vice versa.

1. Since you have only directly connected routes in your simple setup you do not need OSPF.  I suggest that you leave that off for now and let's make sure everything work correctly first before trying to implement OSPF (you probably will not need OSPF unless you have a very large network).  Adding OSPF probably caused some of your initial problems.
2. Port 15 and 23 are not on any VLANs so by default they are on VLAN 1.  I'm not sure if that was intentional or not but in final production network you might want to reconsider that configuration.  VLAN 1 is typically used for management and not for user traffic.
3. Since port 15 and 23 are not configured for VLAN they are by default configured as access ports on VLAN 1 only.  So when anyone try to communicate with these two ports their traffic get routed into VLAN 1 and egress these ports as untagged.  That maybe your intention especially if the DLINK router is not VLAN aware.  In the final production network, that may not be what you want.
4. I noticed that port G4 is configured with VLAN 20 TAGGED whereas all the other G1-G3 ports are configured with their VLAN untagged.  If you configure G4 this way then the PC that is connected to G4 MUST be VLAN aware otherwise it will not be able to understand the VLAN tagged packets.  Also as configured the PC connected to G4 must send tagged packets otherwise the switch will drop the traffic.

Possible problems:

• You mention you had a PC connected to VLAN 10 and one to VLAN 20.  Which ports are they connected to?  If you mistakenly configured G4 to be tagged then if you had a PC connected to G4 then I can imagine that it would not be able to communicate with the other ports.
• Likely OSPF caused some of your initial problems.  Since all your routes are directly connected to this switch (you have no other routers right now) and your network is fairly simple, adding OSPF will unnecessarily complicate the setup.  BTW, since you went back and forth with OSPF settings, have you rebooted your configuration since you made all the changes?  Just in case there are some lingering affect?
• I'm not certain why you would lose connection for such a long time but if ping somehow brings it back I wonder if its an ARP cache problem.  If the system is idle for a period of time then the arp cache would age out and pinging the systems would cause the arp cache to update so maybe its related to that.  This would be more likely related to arp cache on the PC and other servers themselves not on the switch.
• The other thing that could timeout is the MAC learning table on the switch.  If the system idle for a long time the MAC learning table can age out.  If this happens then re-establishing or relearning MAC entries could take a bit of time but not 5 minutes as you described.
• Another thing that could cause some delay would be STP but it should have an effect only if the link goes down then up again.  But good practice is to always enable port-fast (or fast-link) mode on the access ports (access ports are the ports connected to single host or single servers).  Do not enable port-fast on ports connecting two switches (for example don't enable it on the port leading to the D-Link router).

If I can think of anything else I'll let you know.  You might take a look at the above observation and see if anything there would help.  As you experiment further please let me know what else you learn and I'll keep thinking about this problem.

Cuong.

Thanks for the response. I restarted the router and reconfigured ( see current config below ) to get rid of lingering OSPF and any other lingering effects from my fiddling around. Routing happens between computers, but I cannot connect to router interfaces ( or the hosts that lie on the networks of those routers ) directly attached to the 6024. So we are making progress, but I need to get this next bit working if I'm ever going to connect to the internet.

What is working
* on VLAN 10, 20 and 200directly attached computers can communicate within and across VLANS.
* DHCP addresses are served to all directly attached hosts and to those hosts attached via directly attached routers.


What doesn't work:
VLAN 10 192.168.10.0 directly attached to D-Link DI-624 via 192.168.10.1. D-Link IP address is 192.168.10.2. DHCP disabled on D-Link.
VLAN 20 192.168.20.0 directly attached to Sonicwall TZ170. via 192.168.20.1. DHCP enabled for a very small range. This one is critical as this is going to be the connection to the Internet in the production network.

For both these networks, I can ping the 6024 interface (192.168.(10 or 20).1, but cannot ping the routers themselves (192.168.(10 or 20).2), nor any hosts connnected to them, although responses to DHCP requests are forwarded through them to/from hosts connected to them. For example, my wireless laptop picks up an IP from the DHCP server on 192.168.200.0 using the DI-624 (192.168.10.2) as an access point.

Here is the current config of the router...

console# show running-config



Router Configuration
-----------------------------

interface range ethernet g(1-4,15)
spanning-tree portfast
exit
interface range ethernet g(1,3-4,15)
spanning-tree cost 2000000
exit
interface ethernet g2
spanning-tree cost 200000
exit
interface range ethernet g(1-4,15,24)
switchport mode general
exit
vlan database
vlan 10,20,50,60,200
exit
interface ethernet g1
switchport general pvid 50
exit
interface ethernet g2
switchport general pvid 50
exit
interface ethernet g3
switchport general pvid 60
exit
interface ethernet g4
switchport general pvid 60
exit
interface ethernet g15
switchport general pvid 200
exit
interface ethernet g24
switchport general pvid 10
exit
interface ethernet g24
switchport general allowed vlan add 10 untagged
exit
interface ethernet g22
switchport access vlan 20
exit
interface range ethernet g(1-2)
switchport general allowed vlan add 50 untagged
exit
interface range ethernet g(3-4)
switchport general allowed vlan add 60 untagged
exit
interface ethernet g15
switchport general allowed vlan add 200 untagged
exit
interface vlan 10
ip address 192.168.10.1 255.255.255.0
exit
interface vlan 20
ip address 192.168.20.1 255.255.255.0
exit
interface vlan 50
ip address 192.168.50.1 255.255.255.0
exit
interface vlan 60
ip address 192.168.60.1 255.255.255.0
exit
interface vlan 200
ip address 192.168.200.1 255.255.255.0
exit
ip dhcp relay address 192.168.20.2
ip dhcp relay address 192.168.200.5
ip dhcp relay enable
username xxxxxxx password xxxxxxxx level 15 encrypted
snmp-server community Dell_Network_Manager rw view DefaultSuper


OOB host Configuration
-----------------------------

interface out-of-band-eth 1
ip address 192.168.0.250 255.255.255.0
exit
interface out-of-band-eth 1
ip default-gateway 192.168.0.1
exit


_____________________________
>> Default settings:
_____________________________

>> Router Configuration
-----------------------------
>> Service tag: 639V291

>> SW version 2.0.0.01 (date 10-Apr-2005 time 08:28:21)

>> Gigabit Ethernet Ports
=============================
>> no shutdown
>> speed 1000
>> duplex full
>> negotiation
>> flow-control off
>> mdix auto
>> no back-pressure
>> interface vlan 1
>> interface port-channel 1 - 7

>> no router RIP

>> no router OSPF enable

>> spanning-tree
>> spanning-tree mode STP

>> qos basic

>> OOB host Configuration
-------------------------

>> interface out-of-band-eth
>> no shutdown
>> speed 100
>> duplex full
>> negotiation
>> flow-control off
>> no back-pressure
>> exit
console#


Thanks, I hope this is some simple thing to fix. Connecting to the internet is kind of important ;-(=)

Additional observations:

• When there are other routers on the network you must make sure that the other routers have a route back to the 6024 otherwise you can send a message out to the hosts on the route you established to the other routers from the 6024 but there would be no route back.
• If the other routers manage subnets that are not directly connected to this router then you need to setup routing entries on the 6024 that tells it to route to the next hop router for any subnet that are managed by the other router (again remember to add a route back to this 6024 from the other router).  Please see discussion on this thread: http://forums.us.dell.com/supportforums/board/message?board.id=pc_managed&message.id=6193 (look at my last post in this thread).
• On your issues with the D-Link or the Sonicwall, where are they connected?  Your first message you said the D-Link was connected to G23 and your DHCP server on G15 but you didn't have a Sonicwall before.  You seemed to have made some changes since then?
• If you now said that the D-Link is on VLAN 10 and the Sonicwall is on VLAN 20 then I notice a few other possible problems:
• • Port 24 is the only one I see on VLAN 10 so I'm assuming that this is where the D-Link is connected?  Again I noticed that you are configuring it as a member of VLAN 10 only and it is untagged.  I'm assuming that means that the D-Link is not VLAN aware?  If you can ping the 6024 address on VLAN 10 from the other VLANs then you know your packet at least got to the interface correctly and therefore you should be able to ping from the other VLAN to any address on VLAN 10 network.  Meaning that you should be able to ping to 192.168.10.0/24 subnet.  If you could not for some reason ping the D-Link address itself on 192.168.10.2 then it may be something on the D-Link side.  To prove this to yourself.  Connect a PC to the port where the D-Link reside and hard code the IP address to "192.168.10.2 255.255.255.0" and see if you can ping to that PC from all PCs on the other VLANs.  If you can, then check your D-Link configuration.  Remember the route back if needed.
  • Port 22 is the only one I see on VLAN 20 so I'm assuming that the Sonicwall is connected there?  From your configuration port 22 PVID is 1 and the port is a member of VLAN 20 TAGGED.  Does the Sonicwall support VLAN?  Is the port on the Sonicwall back to the 6024 a member of VLAN 20 and is it able to receive tagged packets?  Is it setup to accept untagged packets from VLAN 1 from 6024 (remember your PVID on 6024 for port 22)?  Also any untagged packet returning from the Sonicwall will be on VLAN 1 (again because of PVID).  If you didn't correctly configure the Sonicwall to match the configuration you had on the 6024 then this will not work.

Cuong.

Cuong

Thanks very much for your time and energy in looking at this situation for me. I was on the phone with Dell Techsupport for 4 hours before they decided to send me a new switch. I'm just in the process of setting up now to see if the problem is resolved.

I haven't been active on this for a couple of weeks because I got called away for a server rebuild, so I'm back at it again. Thanks again, just wanted you to know I appreciated your contributions.

T.