Isilon 8.0.0.4 SSD drives at Capacity

Question

Does anyone have suggestions around how to identify the specific data residing on our SSD drives? We have a situation where the SSD drives that are part of our default storage pool have filled to capacity while HDD capacity remained constant. Support suggested that our SSD strategy for snapshots might have been causing snapshot data to reside on the SSD's and possibly not deleted normally during snapshot deletes. They suggested that there are known bugs in our onefs version, 8.0.0.4 for this issue. However, the situation remains that the SSDs are at 98% while the HDDs in the same pool are at 86%. Support has yet to come up with a method to identify which data can be deleted to clear out SSDs. I am willing to delete data, but not randomly shoot in the dark. Any help greatly appreciated.

Here are our current file policy settings for the one and only storage pool that we have:

isi filepool default-policy view

Apply Order: -

File Matching Pattern: -

Set Requested Protection: default

Data Access Pattern: concurrency

Enable Coalescer: True

Data Storage Target: anywhere

Data SSD Strategy: metadata

Snapshot Storage Target: x400_136tb_1.6tb-ssd_96gb

Snapshot SSD Strategy: avoid

Cloud Pool: -

Cloud Compression Enabled: -

Cloud Encryption Enabled: -

Cloud Data Retention: -

Cloud Incremental Backup Retention: -

Cloud Full Backup Retention: -

Cloud Accessibility: -

Cloud Read Ahead: -

Cloud Cache Expiration: -

Cloud Writeback Frequency: -

Cloud Archive Snapshot Files: -

Systems_Team · Answer

Sure, it's possible. One of the compounding factors at the moment is that when I try to run the SmartPools job, what small amount of free space we do have on the SSDs (2%) begins to fill. I do not know why it is filling, but my assumption is that it will continue to fill until the cluster becomes sluggish/unresponsive again because SSDs are completely full. Which brings me back to my original question.. how can I identify and delete whatever is on the SSDs.

--Clint

Peter_Sero · Answer

Suprise ahead...

When a snapshotted file is deleted, it's metadata gets updated, and usually increases by a few bytes.

Once more than 512 Bytes are required, a full 8 KB block will be allocated,

so the disk/SSD space for that file's metadata instantly jumps from 0.5 KB to 8.5 KB, that's a 17-fold increase!

So mass deletions of (small) files can lead to massive suprises in SSD space consumption,

at the time when the "rm -rf" of TreeDelete job is executing.

Now your cluster has the SSD strategy for snapshots set to "avoid SSD", but keep in mind that this (by design)

does not take effect at the time of deleting files. This SSD copy of the metadata is "migrated" to HDD

(or simply removed, depending on wether the SSD copy count as protection or as extra copy)

only at the point in time when the SmartPools job or the SetProtectPlus job is run.

Note that we are talking about snapshots here that are retained and have not yet expired.

So one might see high SSD usage from the time of mass file deletions

to the point when the SmartPools job has finished -- is this something

your cluster might encounter?

Hope it makes sense

-- Peter

Yan_Faubert · Answer

Your issue is bug# 197171 which is resolved in 8.0.0.6. These are internal data structures that cause the SSDs usage to spike and you can't delete data to make the usage go down.

Workaround is to run a 'collect' job but this job can take a long time to complete in large clusters. Ensure SmartPools doesn't run while you are running 'collect'. But be aware that your SSD usage will spike as soon as you run SmartPools again.

RichardRhodes1 · Answer

We just had this problem. The explanation is that all our snapshot meta-data was still on the SSD, even for long ago deleted snaps.  We did the suggested upgrade to 8.0.0.6 and now have a collect running.  Our SSD pct used is slowly falling.  The collect can run for a LONG time.  The guestimate is that ours will run for 2 weeks total.

Systems_Team · Answer

We have completed patching with patch-209299 which we were told would address the snapshot related SSD bugs that are fixed in 8.0.0.6.  Once the patch was applied I rebooted all nodes as directed.  I have subsequently run a collect job and a smartpools job which both finished successfully.  My SSD % usage now sits at 95% while my HDD are at 85%.  So I am still left with the root problem of having a critical alert for a smartpool >90% even though HDD use is 85% and SSD strategy is metadata read acceleration.  Was my cluster simply sized wrong, or is the metadata just large because of the number of files that exist?  I have no idea.  After weeks of having an open sev 1 SR , I have been offered nothing by support in the way of examining and/or identifying what data has filled my SSDs.  Why did it grow?  Which data is new?  What is the nature of the new data?  Floundering.

Peter_Sero · Answer

> Was my cluster simply sized wrong, or is the metadata just large because of the number of files that exist?

Having many many small files takes relatively more metadata than with a fewer but larger files...

With only a little bit more than 1% of the cluster's capacity as SSD,

things can get somewhat 'exciting' with millions/billions of tiny files, say in the 10s of KBytes each.

How many files do you have?

and how many nodes are in the cluster? (node type as above, x400_136tb_1.6tb-ssd_96gb)

-- Peter

Systems_Team · Answer

We have three x400 nodes, and 84 mill files apparently

Peter_Sero · Answer

OK, while I can't offer a method to examine the exact usage of the SSDs,

we can make a quick check for plausibility.

With 84 mio files (directories neglected for now) the average file size is around 3 MB.

Each file takes a minimum of 0.5 KB for (one copy of the) metadata.

Metadata grows in steps of 8 KB when needed, for example for organising snaphots (as mentioned above),

for complex ACLs, and of course, as a file grows: about one 8 KB block per 10 MB file size.

So to get a ballpark here, with 3 MB avg file size, the metadata on SSD should not be more that 0.5 KB + 8 KB

per 'average' file.

That makes about 0.7 TB in total, which is only a fraction of the cluster's 3 x 1.6 TB of SSD capacity.

(If the distribution of files sizes is skewed towards many small files plus a few huge files,

things look a bit different, but not much worse in the worst case.)

I believe that you 'should' be safe, and that you are therefore right something is strange here...

If performance is not critical at the moment, have you considered cleaning up the SSDs

by setting all policies to 'avoid' and running a SmartPools job? (Or even smartfailing

and re-adding the SSDs)?

Another approch to shed light into the matter is to inspect all 'historical' evidence,

like job engine and InsightIQ reports as well as the monthly info events emailed from CELOG,

to correlate the unexpected growth of SSD usage with other cluster activity and events.

Finally, you could pick some (10000) files at random, and run 'isi get -D' on them

to count the SSD blocks by each file (requires some scripting).

This might reveal something, but chances are that the 'dark matter' remains unseen...

-- Peter

Isilon

Isilon 8.0.0.4 SSD drives at Capacity

Was this post helpful?