Snapshot burden on Onefs 7.2.1 cluster

Those 340 snapshots are on non-overlapping directories? And per directory there are just 7 daily snapshots? That shouldn't be a big problem for regular workflows.

But keep in mind that /deleting/ snapshots after expiration can take considerable amounts of CPU and disk IO. Monitor the reports for SnapshotDelete jobs over time (elapsed time <=> LINs deleted and blocks processed) to establish a baseline for your cluster. And as usual, SSD metadata acceleration speeds up jobs, in particular in scenarios with many small files.

hth

-- Peter

Yes Peter, the snapshot are on all différent directory tree! no overlaping!

          snap 1 = /ifs/data/dir1, snap 2 = /ifs/data/dir2, ....  for only 35 directory tree. As usual, some client create a lot of long subdirectore tree.

Question: We are doing snapshot at the head of the tree (ex: /ifs/data/dir35). Is there a big burden because of the snapshot on the cluster, if the customer create a large number of subdirectory with long name and a lot of files under it ???

In my mind, every change under /ifs/data/dir5 is block change and have no effect on snapshot because long directory tree are below ???

And yes we are using SSD for GNA! deleting look like fast. ( Progress:Deleted all of 132124 LINs and 34/34 snapshots; last deleted: 12837; 0 errors.   Running Time: 44s)

Yeah, I wouldn't stress on that. I take snapshots at a very high level of the cluster, and I don't believe they affect performance in the slightest.

Starting from OneFSv7.1.1, maximum number of snapshots total for cluster was expanded to 20,000 by default.

The snapshot per directory stays same to 1024.

Daily snapshot for 35 directories keeping 7 days is much more lower than maximum.

I second that --  just don't expect that the forum will give you a free ticket for your customer. A couple of additional thoughts:

- SSD metadata speeds things up as mentioned, but watch the fill level of the SSD drives carefully. There are steep and high ramps *up* when millions of small files are deleted.  (A deleted but snapshot-retained small file can take up to 17 times more SSD space than the original file until the snapshot finally expires and gets freed.)

- Small files tend to stay unchanged after creation until deletion; unlike database files or log files etc. That's nice for snapshots.

- Reading data from a retained snapshot file while the current file is being modified can be very slow due to overly locking. Kind of sad, but less likely with small files

after all.

Good luck!

-- Peter

Talking about SSD, we use SSD configure with GNA. We have 8 1,4TB SSD drives in our 8 S200 nodes, no SSD in our 4 NL400 nodes. We got Millions of files "/data = 8.8 M of dirs, 125 M files" for 370TB logical used.

Our SSD is only use at 10%. I find it very low and unshure if it work at all for NL400 node because our metadata scan is still very long for our backup true SMB share? the command

Oups! (continue)

The command "sysctl efs.bam.layout.ssd.gna_enable return "1"  as active but ??? proof?? I am not sure to interpret correctly the output of the isi get -DD "files" (see atacht image)  "metadata target n400_144_24gb:56(56)"

Look for meta width device list. But it doesn't appear to have anything listed on yours. What do your file pool policies look like?

This is the filepool Policy how apply to the isi get -DD  previeusly

sysctl efs.bam.layout.ssd.gna_active

tells you wether GNA is actually in effect.

(gna_enable is a settable control, and GNA still could be inhibited when specs are not met.)

In your isi get -DD screenshot, the two lines

*  SSD Strategy:       metadata

*  SSD Status:         complete

indicate that this file has one copy of metadata on SSD, i.e. GNA works for the file which has both data and regular metadata copies on NL nodes.

10% SSD capacity usage leaves you with lots of headroom. Our SSD are about 50-60% filled and it's NOT a comfortable headroom for larger house cleaning activities under snapshots (with that 17x increase I mentioned earlier).

-- Peter

Thanks for your reply,

Is there a way or a command to see the speed différence between a files or a group of files with metadata on SSD and an other group of files not on SSD?

Because my SMB backup scan true a share are very slow and should scan faster on SSD!

Any Idee?

Our SSDs are generally 20-25% full at this point, but we do have a large number of snapshots. Peter, how do you arrive at that 17x increase for snaps?

That's the worst case. Metadata for a small file usually fits in a 512-Byte block, which is the smallest unit allocated on SSD. However when the metadata enlarges, the next allocation unit is a full 8KB block, so it jumps from 0.5KB to 8.5KB, that's 17-fold.

Usually that happens when deleting small files under ten or so snapshots. Imagine a file set whose metadata consumes tiny 3% of SSD capacity, this will jump to a whopping 51%...

-- Peter

Gotcha, thanks.