Unsolved
1 Rookie
•
26 Posts
0
1334
October 25th, 2021 16:00
SQL, Fastcopy and the DD that won't...
Trivia pursuit... without the use of an internet search, what is the most number of files you can have in a single directory? Second trivia question... What are the most number of files you can have on one disk?
Hint the NTFS number and the linux numbers are the same...
Yes, slightly over 4B files... btw slightly over 4B is how many you can have on the disk as well. Whether 1 directory or 1 thousand. /shrug
Now imagine you are traveling through space time... Your SQL admin has created a single directory with over 100M SQL files... If the max you can put in a directory is 4B how many can you backup?
You open the door to FASTcopy... Five days later, you are asking your provider what their definition of fastcopy is?
Avamar/DD 19.4x/DDOS 7.2x (yes actually idpa)
After registry changes we were finally able to get fastcopy to complete... 22 hours later... which is only fast when compared to the previous 5 days when it did not complete.
I believe we need to filter out some of these 100m files... SQL folks, recommendations?
Anyone else attempted to fastcopy a single Ginormous directory with 100M+ files in it?
If yes, what tweaks have you made to your DD, etc?
barry_beckers
393 Posts
0
November 17th, 2021 12:00
Not into avamar(idpa) really, more a Networker person, but what are you trying to achieve or do exactly?
with millions of files, normally I'd expect wanting to do a block based backup, so in effect getting rid of any filesystem overhead and having to deal with millions of files (and client backup index to be updated with these files), but I don't get exactly where or why fastcopy fits into this exactly?
iNetSpy_WBG
1 Rookie
•
26 Posts
0
November 17th, 2021 13:00
I love to load test everything around me... Hardware, software, networks, storage... everything... Part of capacity planning maybe or curiosity as long as it does not effect production.
Block based backups are the way to go... But... I am the new guy digging in trying to see what we have and where we could be better.
What they were doing before - NetWorker Backup SQL 10s of millions of files in one directory. After backups they have attempted run fastcopy for replication (I KNOW). There are so many better ways...
iNetSpy_WBG
1 Rookie
•
26 Posts
0
November 17th, 2021 14:00
There are two articles that discuss the issue, this one (public) and another restricted access.
https://www.dell.com/support/kbdoc/en-sg/000190205/data-domain-rpc-timeout-panics-and-performance-issue-due-to-mdtag-feature
iNetSpy_WBG
1 Rookie
•
26 Posts
0
January 20th, 2022 14:00
I apologize... new to this client... though they have a little bit of everything TSM, Avamar, NetWorker and others... For this problem we are focused on NetWorker... and SQL...
DBAs are insisting their transaction logs must be maintained for 180 days. My understanding transaction logs become useless after at most 14 days.
The result of that decision 100m small transaction logs in ONE directory. We were using Meta Tagging (bad idea, research if you will but I shut ours off.)... Then we started using that LGRE thang... FastCopy went from 4 hours to more than 24 hours IF it completed at all...
How many of you are splitting your mtrees into smaller manageable sizes?
barry_beckers
393 Posts
0
January 20th, 2022 23:00
"How many of you are splitting your mtrees into smaller manageable sizes?"
Wait, wut? Why?
Still no idea why they want to do it the way they are currently doing it using fast copy as nothing is mentioned what was tried and tested before, ruling out regular backup methods?
Not using fastcopy in any way, but depending on what is required, better/other options might be more feasible.
Did you even reach out to Dell at all, specifying what is needed versus what you have in place at the moment and the results to go with it?
As said, normally with millions of files, using NW I'd use BBB block based backups.
Even for 180 days retention, 100M files sounds as a lot? why are those kept online? If you wanna have 180 days of transaction logs, why not simply have that in the backup only and truncate logs whenever they are backed up? But if you don't trust "the backup", which seems to be the case here, as otherwise I don't get what the intend to do even?