Ask the Expert: Integrating Data Domain storage in a TSM (Tivoli Storage Manager) Infrastructure

Nick,

how many customers do you see in the field that use device class=file and actually mount two NFS exports from DD and let TSM write to both mount points simultaneously.

We were looking to get started on the 9th, but I like this question so I will jump in and get things started now.

While that setup isn't widespread, we do see that with some customers.  There are a few different scenarios where I have seen this done:

1. To separate data with different replication needs.  For example if backing up both production and non-production data to the same TSM Server, writing to different MTrees allows you to replicate production data but not non-production data.  We also see the TSM database backups treated differently for replication, where that data is to one (or two) more centralized DDR(s)
2. Where ling aggregation is not possible or desired.  The mount points are then running over separate network ports.  As above, this is often due to differing service levels for different data.  An example would be where you will be doing frequent restores to a SAP QA environment; this would keep the restore traffic separate from the production backup traffic. The real-world benefit here is debatable,as leaving resources unused is wasteful, but it would depend on the overall solution
3. Planning for future growth.  If you know your data growth will require spreading the data to a second DDR in the future, establishing a separate NFS mount on the DDR will make migrating the dataset on the MTree behind that mount point much easier – just replicate it to the new system, quiesce and sync, break the replication and redirect the mount to the new system
4. Improve replication performance (now antiquated by MTree replication using snapshots instead of a log).  The use of multiple mounts to separate directories is used to spread replication load across multiple replication processes, which would help with overall throughput for directory (and pool, for VTL) replication

Is anyone using any of these scenarios, and what are the results you are getting from that configuration?

Welcome to this Ask the Expert conversation. This event has now officially started. Thanks for those who posted their questions ahead of the start of the event and please continue the discussion. Everyone else is welcome to jump in and begin engaging our expert. Let's make this a productive and organize forum for everyone to benefit. Cheers!

Nick,

1) How many TSM customers that you interact with are using VTL versus NFS/CIFS ?

2) How many TSM customers stop using disk storage for primary pool and create primary pool on DD. When we migrated from IBM 3584 to DD we still used a pretty big disk pool on DMX3 to handle the random workload of hundeds of clients kicking off backup jobs at the same time. Also by using disk pool as the primary pool we could easily schedule Data Domain DDOS upgrades knowing that we had enough space in the primary pool to handle Oracle archive logs backup while DD is being upgraded.  What do you see folks do in the field ?

5 years ago there was only one large TSM shop that I know of that was not using the VTL feature on their Data Domain systems, but using devclass FILE over NFS mounts.  Part of their reasoning was the flexibility using NFS mounts provided, where they could easily mount to another Data Domain and direct backups there if the first Data Domain was filling up.  Managing an IP infrastructure being easier than managing FC was also part of the thought process, and lastly it was less expensive to expand their network environment than their FC environment to add the systems in.  It worked well for them at that time, and continues to this day.  They do have a little VTL in their environment as well for a few use cases where it made more sense, but they have been TSM on AIX to Data Domain over NFS since their first Data Domain system was installed. 

Today we still see most TSM environments utilizing VTL, with maybe 20% as NFS only.  We do see some hybrid environments, where a TSM server will have a Data Domain attached as VTL for data storage and also have an NFS mount to another Data Domain, to be used for TSM database backups and an emergency overflow if the primary Data Domain becomes full.  As Data Domain is often being placed as a physical tape replacement, the natural fit is as a VTL. 

As for using a devclass DISK pool in front of the devclass FILE storage on the Data Domain, that is very common, enough to call universal.  For streaming workloads we will see the data go straight to the Data Domain, but anything like your situation with hundreds (or even just dozens) of clients doing simultaneous filesystem backups, having that buffer makes perfect sense.  Your idea to write the Archive/Redo logs to devclass DISK is a great use case as well.

What I have seen is smaller DISK pools being used.  It may be a more widespread phenomenon, but having enough devclass DISK storage to hold the night’s backups is unheard of with the customers I've been working with. They will allow migration to run during the backup window, and in one case have their HIGHMIG set to 0 at all times so data is immediately being written off to the Data Domain after it lands. I think this is mostly the result of the increase in disk performance to handle reads and writes at the same time, and the challenges in getting large amounts of disk (beyond the TSM database and log space) allocated.

I don’t know of a TSM customer using CIFS, but then there aren't a lot of large TSM Servers running on Windows.  I’m sure there are some out there, but I have not worked with them. 

i am surprised that NFS adoption in TSM world is so low. Even though we had plenty of FC infrastructure in place, we wanted to get to something simpler and NFS was it.  And as you mentioned in your example, being able to mount two Data Domains (DD880 and DD890 in our case) gave us the option to change "next" pool for certain management classes when we would run out of capacity on one of the DDs.  We ran TSM on AIX , 10g  layer 2 connection between it and DD890. What also helped my performance on AIX was setting this parameter in dsmserv.opt  ..I was able to get much better throughput to my DD.

DIRECTIO NO

We are no longer a TSM shop,  Avamar has been very good for us

Thank you for the performance tip - that is something that will be helpful to a good number of customers.

We have one TSM customer who is moving from NFS to VTL - their network infrastructure is overloaded, and rather than going through the pain of a major upgrade there, they are offloading traffic to FC.  It's a stop-gap measure at best, but will buy them some time before they have to perform the network upgrades.

I believe we will be seeing more customers moving from VTL to NFS in the future as they are doing equipment refreshes, as that is usually a point where they will review their architecture.  I've yet to talk to a customer who isn't looking to cut costs, and reducing the FC footprint in their infrastructures seems to be a frequent place that is happening.

what kind of dedupe rates do you see on average with TSM/DD customers ? (open systems)

I hate to go with "it depends" but it depend. 

For flat-file data, with the TSM incremental-forever backup methodology, we see anything from 5:1 to 10:1, with the average hovering around 7:1.  Of course, it depends on the data being backed up. Anything in a compressed format (images, video, etc) will get much worse ratios, where other types of data (text files, csv format report data, etc) will get much better. 

For structured data, where you do get into the full/incremental cycles, we see much better ratios, usually at least 10:1 and often 20:1 or better.  And again, it depends on the data being backed up.  I know of a customer with a database full of image files – they are stored in the database itself – which gets limited deduplication and almost no compression, so the ratio there is about 1.2:1.  As that is an Oracle database, we are expecting to see some improvement there with the Application Optimized Deduplication – keeping it simple, the same treatment we give for tape markers but for the RMAN markers. 

When projecting ratios for sizing purposes in a TSM environment, I use 5:1 for flat-file data and 10:1 for structured data, unless we have evidence that the ratios will be different. Those numbers are conservative, and I readily admit that, but I disclaim that with the fact that undersized causes lots of issues, and I can’t remember the last time a customer came in under on their data growth estimates.

fair enough, we were hovering around 8-9x dedupe. These were your typical linux/windows, no databases as they went directly to another DD using Boost or simple CIFS/NFS.

I am out of questions, am i the only one who uses (or used to use) TSM and DD ?

Guess I am odd man out.  All my TSM servers are Windows based and I am using CIFS shares on the DD as recommended by my EMC reps (as opposed to VTL).  One of my TSM servers has just under 1000 clients on it.  They write directly to the DD; i.e., no DISK pool in front of the DD FILE pools.  Only time I seem to need that is when I need to down the DD.

Anyway, on to my questions.  Right now I only have a TSM FILE-class disk pool configured on the DD.  When EMC helped me set this up they had me create an MTREE and share it out to the TSM server.  In the file share they had me create three folders:

• DDPOOL01

• DRM

• TSMDBBACKUP

The DDPOOL01 folder is where my copygroup for the disk pool points, but I never received any documentation for configuring DRM and DB backups using the DD and these other folders.  I know what they should be used for, and I know how to set up DRM and DB backups to use them, but I am looking for EMC documentation that describes setting up the TSM server for BMR when using a DD appliance.  That document should spell out the configuration of these folders and their copygroup setup.  Anyone seen a document like that?

Now I can say I know of at least one large TSM environment running on Windows.  I knew someone had to be doing it, I just had not worked with one until now.

I’d be interested to know what your mountlimit is for the deviceclass pointing to DDPOOL01 on the Data Domain – with that many clients I’m wondering how many write streams you are allowing to the Data Domain system, as well as what model it is. 

For the directory setup on the Data Domain MTree, I’m unable to find a specific document highlighting the design plan, but I can explain the purposes of the three directories, along with some general comments about them:

• DDPOOL01 – as you are already using it, this is for where the backup data is written, based on the devclass referenced in the backup copy group
  • When replicating this data, don’t forget to have a reusedelay set to the same as your database backup retention on any storage pool stored here.  With replication, the replica destination is acting as your copypool, and is being kept synchronized with the primary pool, so you need to make sure you aren’t deleting volumes from it that that would be referenced in a TSM database backup used for a recovery
• DRM – this directory is for the DR Plan files.  When you run the PREPARE command in TSM, the planprefix parameter should point to this directory
  • As this is on local disk, you’ll periodically want to clear out old plan files as TSM will not managed them for you.  They are going to compress very well (being plain text), and you will also see global compression (deduplication) , so they won’t take up much real space on the Data Domain system, but you won’t want to have to find one plan file out of thousands if you do need it
• TSMDBBACKUP – this is where you would want to write your TSM Database backups.  If you haven’t already configured one you will want a devclass pointing to this location.  You could also write the TSM DB backups to DDPOOL01 using that devclass, but keeping them separate makes it easier should you need to look for a specific backup.

To perform a recovery on the destination side, remember to make a copy of the data to use for that exercise.  For a recovery test, the replication destination is ReadOnly which will cause issues with, and TSM will have issues with that, and you would not want to try to restore from an environment that is actively receiving updates via replication. In a true disaster, you would still want to keep the replica data in place until the recovery is complete, just in case something bad happens during the recovery (like all volumes in TSM being deleted…). 

Great news! Our SME Nick_Cassimatis  is extending the opportunity for our users to post their questions. He has kindly requested to postpone the end of this event for another week. So continue posting your questions on this thread until February 27. Cheers!

My question is a probably "it depends", but I can't seem to find much info out there.  We've got three TSM servers that all hold about 300 (per TSM server) mix of Windows/Linux nodes, just doing file system backups.  We've got those TSM servers using NFS to the DD, when creating the DevClass and Storage Pools, are there any suggested settings for particular things, main ones I've been concerned about are MaxCapacity on the DevClass, which I've seen mentioned as low as 2GB to as high as 400GB, was wondering if there was a basic suggestion there or not, and any specific tips on the Storage Pools as well.

Also, we don't have replication set up yet, not sure when that will be; can you use tape for a copy pool for the DD, do you have anyone that does that who may not have replication licenses?  Thank you much for any answers you can provide, very glad I stumbled across this forum.