DDVE space overhead and leak

I may be seeing similar issue as the OP.  I added only a minimum 200gb disk to the ddve to use for my filesystem.  I didn't look too close after adding and assumed I had 200gb to work with.  I setup an integration of networker using ddboost with a 100gb test client and was very surprised to see I had filled the filesystem at 75gb.  Looking at the filesystem tab I see the following sizes allocated:

/data:post-comp 76.5 GiB

/ddvar 49.2 GiB

/ddvar/core 158.5 GiB

Based off my existing 5.5 Data Domains, "/data:post-comp" seems to represent all the real usable filesystem.  I would expect that /ddvar and /ddvar/core would have landed on the internal disks defined by the ova template.  Where did the remainder of my 200gb (186 GiB) go or what is the behavior here?

Hi, in response to your capacity question.  the space you are seeing is normal for the system overheads on the 1st data disk.

When a DDVE is deployed for the 1st time we see 2 disks automatically configured.  These two disks are seen as a 250GB system disk, and a 10GB NVram disk.

We do not have any storage for customer Data configured on initial deployment of DDVE2.0.

You will need to add a minimum of 200GB storage for the 1st Data disk presented for customer usage. This 200GB will be broken down in the following way to give you usable storage:  200GB – 120GB for system disk – 5.6% for Raid on Lun = Usable storage.

Math Example:

200GB – 120GB = 80GB – 5.6% = ~76-77GB

300GB – 120GB = 80GB – 5.6% = ~169.92 – 170GB

(The 120GB on the  1st added Data Disk is for DDOS to make backup copies of Lic’s /reg / core / logs etc on the ext3 partition – this is the same as when we have ext3 partition on the 1st Physical shelf of a standard Datadomain.)

NOTE: After the 1st Data disk has been added and the filesystem is created, you can add further increments of 100GB Data Disks.  You will see only the 5.6% overhead for Raid on Lun.  We only take extra capacity for the system requirements from the 1st Data disk.

for further in depth explanation of the 'real' utilization pls see: https://community.emc.com/thread/225341

how are your cleaning figures showing now after some more ingest & cleaning?

Thanks, Ben.

With regards to total usable space on an instance of DDVE, as mentioned by Ben, you need to be aware that a significant chunk of the first data disk will be taken for internal metadata so, for example, if you add a single data disk of 200Gb in size you will actually only have a usable file system of ~76Gb in size. Please refer to the link Ben added for more information.

With regards to small files - this is a known issue with DDRs in general. Basically they are designed to hold a relatively small number of large files (commonly the type of files which backup applications create) rather than a large number of small files.

To explain this further - when DDFS de-duplicates data it does this by splitting incoming files into variable sized chunks of 4-12Kb (with each chunk being a segment). We then check each segment to see if we already have it on disk and if we do its a duplicate so can be dropped from the incoming file and essentially replaced with a pointer to the existing data already on disk.

When you write very small files (i.e. < 4Kb) these are smaller than our minimum segment size so DDFS will:

- Pad the files to 4Kb (minimum segment size)

- Not de-dupe them

- Will compress them (however compression ratio of a very small amount of data is generally poor)

- Write them out to disk

When cleaning runs we try and reclaim some of the space taken by small files by looking for duplicates however, in general, overall compression ratios and so on will suffer if your data set consists mainly of small files.

In addition to the above there are performance issues caused by small files - in previous versions of DDOS these caused slow cleaning (however this is not an issue with DDOS 5.7/DDVE) and in all versions can cause poor directory lookup performance (i.e. browsing directories via CIFS/ls vi NFS) and poor replication performance. The reason for this is beyond the scope of this forum suffice to say that storing huge numbers of small files on a DDR is generally not a good idea (and not really what DDOS was designed for).

With regards to why writing 1.2Gb data caused file system utilisation to increase by 15Gb - this is hard to say without more information about the system. In addition if you are deleting files and not getting as much space back as 'cleanable gb' indicates this might be because:

- Cleanable Gb is an estimate (again too much information to go into this in depth here)

- The physical data for files which have been deleted might be being held on disk by files written after the files which were deleted

- There might be file system snapshots which contain copies of the deleted files causing their data to be held on disk

- There might be replication lag causing deleted files data to be held on disk

In all versions of DDOS the cleaning functionality has been found to be generally robust so if you really are deleting a large amount of data and not getting any space back by running clean I suspect its due to one of the above reasons.

I hope this helps.

James, thank You for in-depth answer.

I noticed that leaked space starts reclaiming to cleanable space during write operations of next dataset. This behavior match internal DD processing that You describe above.