Backing up Sun 7000 via NFS using Networker 7.6.1

Hi,

Bit of a long shot, but this is something you could check:

esg117335: NMSAP backup is slow because of ZFS prefetch after appyling a Solaris 10 patch

-Bobby

yes i had tried that fix, no joy

Have you tried just copying a file from across the NFS mount.  Is that really slow aswell or is the problem specific to the backups?

-Bobby

yes  copies\rsync's the time is the same or better on the problem system.

I ran some tests (below) and saw an increase in metadata reads, so at this point it looks like it is having trouble with small files and metadata.  This is not for sure but it is the only trace of problem that I have seen.

time find . –print > /dev/null

time /usr/sfw/bin/gtar cf /dev/null .

Hi,

Hmmmm - so backups are really slow but file copy is fast.  Another test you could try is to use the bigasm test in NetWorker which will isolate whether the issue is with reading from the disk or if it is somewhere else.  You can find details on this test in the Performance Optimisation Guide page 56

or more specifically on how to do it here:

https://community.emc.com/message/345160

or

https://community.emc.com/message/340669

-Bobby

check that you haven't got any hostname resolution issues between the two hosts. That can really slow down NFS.

/tony

Thanks, I probably will have some questions later. We are going to attempt to connect the Sun 7200, which is iSCSI, to the VNX 5300, which is FCoE, via a Cisco Nexus switch. Fun times will be had! :P

Thanks for the reply.

I have struggle through the 7000 setup.

The protocol as far as I know is a zfs send\recieve.

I don’t know how experienced you are with the 7000 but they can be a pain, if you have a questions I open to helping.

Just thought I'd add to this post, because I am also migrating from a Sun ZFS Storage 7000 (the 7200 to be exact), and maybe someone else will also find this in a search.

The "Rsync" service on a Sun 7000 box is actually Sun's proprietary protocol. It is not the universal standard rsync. They rewrote it specifically for transferring between two Sun 7000's.

I learned this from one of their engineers when I tried to use rsync to a different device, and it wasn't working very well. It's one of the many reasons we're switching from Sun/Oracle..

Yes that does sound like fun.

My best recommendation before you do anything is move to the newer patches, not sure if I trust the latest patch, but the second latest patch according to sun is stable.

Not to scare you but I have horror stories about these servers in general and oracle support that will crack you skull.

The 7000 was one of the signs of a failing company.

I think everyone has horror stories with that system. Sun had absolutely no idea about NAS, but they wanted to add unified storage to their product line, so they found some spare parts, added a million (unfinished) features and tied it all together with tape.

Their most hilariously bad decision was adding software-based deduplication in a patch, because everyone else was offering it. It was so bad they had written warnings and recommended you don't use it.  Turn that on and you might never see your data again.

We have used it for 2 years for our virtual servers and file shares. It never worked right to this day. Replication never works. Analytics use up massive resources if not turned off. Constant crashing, worthless or nonexistent logs, unexplained slowdowns, SP lock ups cause the entire thing to crash...  I'm scared to look at it wrong because it might self destruct.

Oh, did you want good performance? You'll need Logzilla SSD drives, for a ridiculous $10,000 each. Good luck explaining that purchase.

It's a shame because ZFS is an excellent file system. And the performance analytics tools are extremely detailed. But it simply fails at being "network storage".

Anyway, I think we are on the second to latest firmware. I know they just released an update recently. Hopefully it won't fall apart before the VNX is up and running.

Nice, great reply. I think we are on the same page. But now you have scared me!!!

I have railed at Sun Rep and Support for the last 8 months for having released this half baked product. It amazes me how they try and defend the product and oracles horrible support, with a smile.

Anyway I am stuck with the 7000 for the next few years, if I don’t take a chainsaw to it first.

The dedup is a good tip. If you have any more let me know, some of the crap that you have to put up with on these servers are better avoided like the dedup. Glad I never tried it.

thanks

Well you're in for some good times.   Here's a few tips I can think off the top of my head...

-Make sure you delete all unused analytics datasets because they use up tons of drive and memory resources.

-Add it to your Windows domain if you have one, to make permissions much easier to manage with active directory.

-If you don't have readzilla/logzilla SSD drives, consider getting them. (And you'll need two, in mirrored RAID, because we had a failed logzilla recently, which would have been a nightmare without the backup). Otherwise, you're using the built-in memory write cache, which will cause data loss if it crashes.

-Set up a dedicated switch and subnet for iSCSI traffic, and turn on jumbo frames (9000 bytes) packets. You'll need to configure that on all the interfaces of the Sun, the switch, and the VM hosts.

-Replication is not block or file level, so even when you turn on "Continuous", it's actually just uploading entire snapshots on a schedule.

-Call-home is useless. They never responded to any tickets our unit created via call-home, in fact they usually just close them. Also, when you upload a support bundle, copy the bundle's filename. Support will always ask you for it even if you included it in the ticket.

-Watch the disk OPS in the Status monitor. Ours is average 7,000 - 9,000 OPS, which is already too much for it to handle. When it peaks over 20K, everything, including the management page, becomes unresponsive. It's very easy to overload the machine, which will cause it to either slow down to a snails pace and cause your VM's to time out and crash, or cause the NAS itself to crash.