How do YOU back up your Isilion?

We use SyncIQ to replicate the data to our offsite data center.

You could use a client to mount up the /ifs via NFS and back that up just like any other client.  I'm not a big fan of NDMP.

The reason I suggested NFS was simply because that's we use our Isilons for - almost 100% NFS v3 - so an NFS-based backup strategy would catch it.  If you're strictly an SMB shop, backing up via NFS would be pretty foolish as Chris pointed out.  If you're 100% SMB, then backing via SMB shares might be an option.

I still prefer SyncIQ for our environment simply because it's not practical to walk petabyte file systems.    That's also how I found a few SyncIQ bugs

There as a presentation by Dr. Stefan Radtke, CTO EMEA about Backup Concepts. It has some nice comparison between backup possibilities.

It is available at the following link:

http://germany.emc.com/collateral/customer-profiles/isilon-backup-concepts.pdf

some deeper informations are available here:

http://stefanradtke.blogspot.de/2014/09/challenges-and-options-for-filesystem.html

The OneFS NDMP implementation does a pretty good job crawling large hierarchies,

due to extensive prefetching along the snapshots it processes. NFS/SMB based backup

clients would need some manual integration to use snapshots, and still couldn't

do that level of prefetching. Moreover, OneFS 7.1 NDMP can do snapshot-based diffs,

which are really fast.

If you can increase the number of concurrent NDMP jobs as your cluster is growing,

it can go quite far. For disaster recovery, we do nightly diffs of ca. 50 shares

containing 150 Mio files, half of which have metadata on SSD (3 X + 5 NL nodes),

with up to eight parallel NDMP jobs. And that's with 7.0.2 old-style crawling diffs!

So far we see no reason why linear increase of data, node count *and* NDMP jobs

should not keep working work for us. Partly because the number of our shares

(or backup-configurable subfolders) will increase so that we can run more jobs,

partly because tape (LTO) technology more or less keeps up with the growth of

individual shares/folders. YMMV of course.

Happy backing-upXXXXXXXXXrestoring!

-- Peter

Interesting Read, I need to read up on the change-list feature

ps, the pdf has "EMC CONFIDENTIAL—INTERNAL USE ONLY" on every page, I hope it's ok to share It was a good read.

For some of us, NDMP really is a good choice.  We configured an accelerator node, attached via FC to an LTO tape library.  We rely on 30 days snapshots for day-to-day user file recovery, then run weekly and monthly fulls to tape.  This has been an extremely cost-effective solution for us, and still meets our operational goals.

The only thing that would increase our solution would be to purchase a partner cluster, configure it to keep 60 days of snapshots and replicate to it via SyncIQ.  At that point, I would probably move my tape library to a third location, and backup the replication target to tape.  That way, I have instant recovery from local snapshots, longer-term recovery from the DR cluster, then archival recovery from geographically-distant tapes.

Full disclosure:  I'm a fan of the NDMP protocol, regardless of it's inability to restore across vendor solutions.  I manage several NAS products, some from multiple vendors, and NDMP lets me share resources (tape) where I need it. We've been able to provide three-way NDMP backups and restores for other organizations on Campus, who didn't have much experience with NDMP.

Karl, are you already using snapshot-based diffs for NDMP?

I have tried them out and found, they are fast but still we can't use them

for most of our production share because the full (= NDMP level 0) snapshots

will get too large.

You wrote that you retain regular snapshots for 30 days,

that should also cover the full backup cycles, right?

Cheers

-- Peter

I think the math is not working out in your favor.  Let's assume that you're going to be able to take 5Gbps of backup traffic per node, all day long.  And that's really, really stretching it.  That's 25GB per second.  That's 90TB per hour or 2PB per day.  I don't think there's any chance you'll get that traffic out of our cluster all day long and certainly not out of a cluster full of NL400 nodes.  You don't want that amount of traffic on your primary cluster, along with the replication traffic, or you'll never get any real work done.

Getting 300TB of replication traffic out of your cluster onto your NLs is hard enough.  Pushing that much data to tape is going to be impossible (and expensive to build out).  The only shot I think you've got is to do backups to disk and continuous incrementals and even that first copy is going to be pretty ugly.

If you do get it working, I'd really like to see what you've built.

Jim

a couple of more thoughts.

300TB/day is 5TB/day/node or 70MByte/s/node.

Yes, the X nodes need to spare a bit headroom,

but only a bit, from their HPC load to sustain this.

If those 70MByte/s/node arrive at the NL nodes,

these should be able to pass it on to the tapes, throughput-wise.

If you can keep enough tape drives busy in parallel

(at 100-200+ MByte/s/drive), that is. It depends on the

directory layout(!) and typical file size, of course.

10GE on NL nodes will be fine, and easier

to manage than FC and backup accelerators.

Quite a few tape drives will be needed, but hey,

you scale out the storage, so you scale out the

backup infrastructure...

300TB/day and max 90 days retention will result in

up to 27PB -- yes, there are tape libraries that large available.

300TB/day change rate on a 3PB cluster -- that's 10%/day.

I like that.

Because it indicates a good chance for having mostly

short-lived data on the cluster

If such data can be isolated for a dedicated backup strategy,

then consider doing incremental-forever (NDMP level 10) runs

with a fixed retention interval(-!-)  This is an uncommon

method with NDMP because one would loose "persistent"

data from the backup, but for "scratch" data in controlled

workflows (which you might have) this works.

Remaining, more persistent data, can be much less

than 3PB and thus get backed up with normal

full-cumulative-incremental cycles, if (again if) the

directory layouts permits sufficiently many parallel jobs.

Also, partial full backups can run in different weeks.

It is a big challenge, but supposedly big fun, too. The more

knowledge about existing workflows and data life cycles

can be leveraged, the better.

Or: Try to inquire the price tag for the new Isilon HD400

high density nodes. I have no idea on the pricing yet, but do hope

that the price per TB will be "somewhat" lower than with NL400...

Hope it helps

-- Peter

Technically, no we're only running full backups to tape as NDMP level 0, so no diffs or incrs at all.  Originally, we put a pre-script line in Netbackup to have it create a snapshot for the duration of the backup, but eventually, we chose to simply point it at a particular snapshot and just let it run.  For us, we could let each weekly full backup to tape take up to seven days - we already rely on snapshots for our day-to-day restores.  The tape backups just give us offsite DR.  Again, a partner cluster with more snapshots would be even better for us.

Thanks!

Karl

IsiBackup allows you to make backups of all configuration items available through the GUI of OneFS.   IsiBackup does not store the actual data (files and folders) that reside on the file system.  IsiBackup focuses on the systems configuration elements, such as: access zones, smb shares, nfs exports, quotas, network configurations etc.

IsiBackup allows you to keep track of all the configuration changes that have taken place on the cluster between calls to IsiBackup.  It can be launched automatically at regular intervals and it will send notifications by email when configuration changes are detected. 

This software can also restore specific Isilon configuration items, provided that they have been previously saved by IsiBackup.   The restoration of configuration can be performed on the same cluster or another cluster.  For example, it is possible to duplicate elements from one cluster to another. 

Networks shares and quotas can also be restored on a different access zone as needed. 

Here's how to restore thousands of quotas with IsiBackup in less than 90 seconds: https://goo.gl/w7Ns7Z

For information, or for a free trial, please contact info@gallium-it.com.