SSH Sessions repeatedly hang on 3348.

Sorry, you're right. It is a 3448.

Still does the same thing with 112

Does it happens only on an SSH session or does it happens also when you simply telnet into the switch?  Also does it happens only when you have all five SSH sessions or can it happens when only one SSH session is active.  How easily is it to reproduce?  Do you have to do "show spanning" repeatedly over many times to reproduce it?

Any info that might help me reproduce the problem is appreciated.

It happened to me while using SSH, though I haven't tested it with either the serial console or telnet.

Reproducing it is quite easy.

It doesn't seem to be consistent in the number of times you have to show spanning-tree. The first time it happened was while configuring some spanning tree settings, so in that case, it was only run a few times between other commands. When I first tried to reproduce it it took several tries (maybe 5-6), though sometimes it may take 20 or 30. You should be able to reproduce it easily by just repeating the command many times.

If you don't have any luck with that I can send you my config, though I don't know that that will be necessary.

I have tried with both SSH and regular telnet sessions.  I have executed the show spanning tree operation more then 50 times on each type of sessions.  I also tried with multiple telnet & SSH sessions active.  I'm using SSH on cygwin and on Linux.  No luck reproducing it.

Can you give me more information please.  When you said it locked up on you, does it hangs only the given SSH session so that you can access it with another SSH session?  If so maybe something to do with your particular SSH tool.  What are you using?  Is it SSH that comes with Linux or something on windows?

Does the console show any message when a session "locks up" on you?  Any other observation that might help?

Cuong.

I think I figured out the problem.  It seems that some SSH clients on windows will seem to lock up when use with the switch.  It maybe that the performance on these clients are so slow that they look as if they lock up when executing commands that result in a lot of data being returned (such as "show spanning").

I have downloaded and tried a number of different SSH clients on Windows and I find that there are really three types:

• SSH clients based on OpenSSH which requires cygwin - this is what I used normally and these works fine.
• SSH clients based on Putty.  These seems to have some performance problems when connected to the switch.  There are a number of these types of clients all based on Putty including the one in "TTermpro" (look here for some information on these types of clients: http://www.openssh.com/windows.html).
• Alternatively you can also use SSHClient which is not based on OpenSSH or Putty and seems to have the best performance of all SSH clients available on Windows.  See here for information on this client: http://ecs.rutgers.edu/ssh_windows.php3.  The latest version of this client is SSHSecureShellClient version 3.2.9 found here (ftp://sunsite.unc.edu/pub/packages/security/ssh/).

Also when these slower clients are killed they do not release the session which cause the problems you notice where the sessions are locked open.  In fact if you are really patience these clients do eventually response just take a VERY long time.

I suggest you use the OpenSSH based clients on windows or the SSHSecureShellClient for better performance.

Cuong.

I have been using OpenSSH in linux on more than one seperate machine.

"OpenSSH_3.8.1p1 Debian-8.sarge.4, OpenSSL 0.9.7g 11 Apr 2005"


The first time the problem occurs the session disconnects "Connection reset by peer" as in the log in my original message.

Subsequent attempts to do show span (or many other commands) will hang the session. (no response within several minutes) the session seems to have hung completely and stays open on the switch even after killing my client and connecting again.

Here's the config, incase that's helpful.

As I mentioned before, I've tried it on 2 switches with the this config, and both had the same problem.




nick@gorf:~$ ssh s1-c8


s1-c8> en
Password:********

s1-c8# show running-config
spanning-tree mode mstp
interface range ethernet g(3-4)
switchport mode trunk
exit
vlan database
vlan 30,50,70
exit
interface range ethernet e(1-48)
switchport access vlan 50
exit
interface range ethernet g(3-4)
switchport trunk allowed vlan add 50
exit
interface range ethernet g(3-4)
switchport trunk allowed vlan add 70
exit
interface vlan 30
name colo
exit
interface vlan 50
name colo2
exit
More: , Quit: q, One line: interface vlan 70
name management
exit
interface vlan 70
ip address 10.100.32.71 255.255.255.0
exit
ip default-gateway 10.100.32.254
hostname s1-c8
line console
autobaud
exit
line console
speed 115200
exit
aaa authentication login default none
line ssh
password 00000000000000000000000000000000 encrypted
exit
enable password level 15 00000000000000000000000000000000 encrypted
username \ password 00000000000000000000000000000000 encrypted
ip ssh server
snmp-server community Dell_Network_Manager rw view DefaultSuper
More: , Quit: q, One line:





Default settings:
Service tag: GYWYK71

SW version 1.0.0.106 (date 01-Jun-2005 time 15:26:21)

Fast Ethernet Ports
==========================
no shutdown
speed 100
duplex full
negotiation
flow-control off
mdix auto
no back-pressure

Gigabit Ethernet Ports
More: , Quit: q, One line: =============================
no shutdown
speed 1000
duplex full
negotiation
flow-control off
mdix auto
no back-pressure

interface vlan 1
interface port-channel 1 - 8

spanning-tree
spanning-tree mode STP

qos basic
gorf%

I've loaded a configuration very similar to yours.  The difference is that I'm using VLAN1 for management and I adjusted the IP address to fit my network, and of course I changed the login password.  Otherwise the configuration is very similar to what you have.  I have executed the "show span" on this switch for more then 50 times using both SSHClient on windows and OpenSSH on Linux.  I have not had any problem.  I also tried using telnet with similar result.

When you run this, does it seems slow on any other command?  Meaning that do you experience a general slowness that might be attributable to having to encrypt/decrypt using SSH?  Or do you normally see everything working with good performance and experiencing a problem only on the "show span" command?  Also does it "lock up" when it is "gathering information..." to display the spanning tree or in the middle of dumping the data to the screen?

Can you attach a console to the serial port of the switch while executing your command to see if there is any messages printed to the console that might help to diagnose this problem.

At this point I'm really at a lost.  I cannot reproduce the problem on my network and I'm not sure what you are seeing.  I can't figure out whether the problem you are experiencing is SSH related or something else.  If you could reproduce it using only telnet then it might help to eliminate SSH as the possibility.

Cuong.

if you look back at my original post, you can see the answers to most of your questions.

As far as me doing testing on the console or with telnet, I think you should be able to handle that given the information at hand.

I'm not sure what you could be doing so differently from me that would prevent you from reproducing the problem as both myself and a co-worker of mine independantly ran into this, both using our own computers, with nothing common between them other than perhaps a similar config.

Thanks.
-Nick

I tried duplicating the issue using a 3448 that provides link to all of our lab systems (different lab than Cuoung) and was also unable to find a problem. I issued the "show span" (using both "terminal datadump" and "no terminal datadump" 100 times and experienced no lockups. I also tried hopping through more than one SSH tunnel by using PuTTY to connect from my Win2k3 server to my RHEL4 U1 server and then SSHing into the switch, but saw no difference.

3448 fw - 1.0.0.112
openssh-3.9p1-8 (RHEL4 U1)
3448 uptime - 24 days

Is the problem intermittant? If you reboot the 3448 does it appear to work for an unknown amount of time? What is the uptime for one of the switches on which you've seen this problem?

 

The switch was a little more stubborn this time, but still broke.

logs are at http://arx.ca/dellswitch/

The order things were done, just to clarify.

ssh.log, where I try to config the switch, but my editor won't let me open my terminal prog in its record mode.

minicom.log So, I open the terminal in another window, do the basic config STUFF (jeeze your filters are picky. Maybe they should just turn **** to stars rather than making me fix it) to be able to login via ssh. (This is a brand new switch, never used)

ssh2.log I was getting tired of waiting for "gathering information....." all the time, since the switch was being more stubborn than usual, so I opened another session. good timing I suppose as it broke almost immediately. (note that I'm pretty sure multiple sessions are NOT required to break it)

It took about 15 minutes for me to break it this time, though it's usually not that stubborn.

all 3 files were left open, so you can see the connections (and disconnections) on the minicom window while the ssh stuff is breaking.

don't worry about all the ^H and stuff... I can't type. Clarified some of the more confusing parts, but I'm pretty lazy, and have more important things to do right now.

l8rs

-Nick

Message Edited by nickw- on 10-14-2005 11:51 AM