8024F Upgrade from 3.x.x to 4.x.x

We've run into this issue numerous times with at least firmware 3.1.3.9 and the latest 3.1.5.13. It seems to be an obvious bug in the v3 firmware. I prepared a bug report for Dell support, but haven't sent it yet. Hopefully it will get past their first line without a quick "use firmware v4" response. And yes, v4 (tested on 4.1.0.19) seems to work fine with scp/sftp.

Here's the report. Attachments won't be available, of course. They are described, though.

# Start of report

Bug report for Dell PowerConnect M6220.

System: PowerConnect M6220, 3.1.5.13, VxWorks 6.5

Problem summary:

Using scp or sftp to upload files from the switch CLI or web

interface, hangs the switch management indefinitely.

Steps to reproduce:

Power up factory-fresh Dell M1000e with one M6220 in slot A1. For

completeness, this particular system was also set up with a M6220 in

slot A2 and 16 M610 blades (powered down). No stacking between the

M6220s.

Apply a static IP address on switch-1 using the CMC command:

racadm setniccfg -m switch-1 -s x.x.x.2 255.255.255.0 x.x.x.1

Upgrade the firmware on switch-1 from the factory default 3.1.3.9 to

3.1.5.13 using the web interface. HTTP transfer method. Reboot the

switch with the 3.1.5.13 image set to activate on next boot.

Using either the web interface or the CLI, start an SCP or SFTP upload

of any file to an external host. Here's how it looks in the CLI:

====START====

console#copy startup-config scp://testuser@x.x.x.3/foo.cfg

Remote Password:******

Mode........................................... SCP

Set TFTP Server IP............................. x.x.x.3

TFTP Path...................................... ./

TFTP Filename.................................. foo.cfg

Data Type...................................... Config Script  

Source Filename................................ startup-config

Management access will be blocked for the duration of the transfer

Are you sure you want to start? (y/n) y

SCP Configuration Script transfer starting...

0 bytes transferred...

====END====

What seems to happen at this point, is that the switch connects to the

remote host, then starts to do the SSH protocol handshake. Once the

remote host has sent SSH2_MSG_KEXINIT, the switch should also send its

SSH2_MSG_KEXINIT back to the host. Instead, the switch goes into an

infinite loop sending empty packets.

At this point, the CLI becomes unresponsive. The web interface lets

you connect to it, but it returns the scp/sftp progress html in all

frames, rendering the web ui useless and impossible to log in to.

To recover from this situation, the only solution so far seems to be

a switch reset through the CMC: "chassisaction -m switch-1 reset"

Attachments and their descriptions:

m6220_scpfreeze__show_tech-support.txt: Tech support output from

switch, after firmware upgrade, before copy command is run.

m6220_scpfreeze__copy_command.txt: Text output of scp command that

fails. The console and web ui freeze right after this output.

m6220_scpfreeze__postfreeze_loginscreen.png: Screenshot of the login

screen after the freeze has occurred. Notice how every frame and every

object on the page is replaced by the scp/sftp progress html.

m6220_scpfreeze__packet_capture.pcap: Packet capture showing how the

switch and remote host do the SSH handshake. The switch fails to

complete the key exchange, and instead goes into an infinite packet

loop. The capture file is cut short to avoid a large file size.

m6220_scpfreeze__sshd_debug.txt: Host-side sshd debug output.

Corresponds with the packet capture, and shows how the handshake

stalls after the host sends the key exchange init.

m6220_scpfreeze__v3.1.3.9_stacktrace.txt: A similar but older

occurrence of the same problem on firmware 3.1.3.9. This one might

prove interesting, as it has a stack trace in the output. It is

unclear if this 3.1.3.9 crash caused the packet loop we see with

3.1.5.13. The switch hung, and failed over its stack manager role to

the other switch in the stack.

# End of report

Are you attempting to do this remotely or do you have physical access to the swtich?

It's done by logging in via ssh to the m1000e cmc, then connecting to the oob switch management console.

On a side note, I reported this issue a few days ago to Dell support. I'll post updates here as I get info from them.

This is absolutely fantastic, thank you for going to all the trouble of tracking down this bug.  I was beginning to worry that I was suffering from some sort of specific, Dell hating delusion. :)

Although, i wanted to point out that I am still having scp problem as of version 4.1.0.19.  The behavior is identical to version 3.x.  I haven't packet sniffed or sshd debugged it as i've been fairly busy, but i'll give it a try when i get some time.

Yup understand that...  Can you log into your switch and do a show run.  Make sure your not running simple switch mode?  There are many features that have been removed from the Full OS to the simple switch mode...

So you still see the problem on 4.1.0.19. That's very interesting, and I bet Dell support are interested too. Could you do the packet dumping/testing/debugging to see what might go wrong?

Here's the answer from Dell support, where the conclusion is that v3 won't see any changes or bug fixes at all:

===QUOTE START===

Version 3.x.x.x of software for all blade (M6220, M6348) switches as well as PC70xx and PC80xx series switches is in the END OF DEVELOPMENT status.

All of these switches have v.4.1.x.x available and this is the only version which will be developed.

Long story short there are no further plans to release any v.3.x.x.x even for bug fixes.

===QUOTE END===

Update?