Skip to main content
Start a Conversation

Unsolved

Z

Z_A_P_a0ce3f

1 Rookie

 • 

3 Posts

89

November 23rd, 2023 15:12

ME4 scheduled snapshots issue

Hello,
We have an issue with a ME4 system and scheduled snapshots. There are a number of volumes and most of them are configured for multiple snapshots.
For some time now not all scheduled snapshots can be executed. The error logged is:


Event:
Scheduler: Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large. (task: Vol0011-xxx, task type: TakeSnapshot)
EVENT ID:#A60954
EVENT CODE:362
EVENT SEVERITY:Error
EVENT TIME:2023-11-21 02:31:07
Midplane Serial Number:xxx
System Name:ME4024
Bundle A Version:GT280R011-01
MC A Version:GXM280R013-30
SC A Version:GTS280R10-29
Bundle B Version:GT280R011-01
MC B Version:GXM280R013-30
SC B Version:GTS280R10-29
Additional Information:unknown
Recommended Action:unknown

The system has plenty of storage space left. If we reduce the number of stored snapshots (manual deletion), the error does not occur for a while. We know that the snapshot space can be too small.
But too big?

Thanks.

DELL-Young E

Moderator

 • 

3.9K Posts

November 23rd, 2023 23:25

Hello thanks for choosing Dell.

This schedule- What is the schedule setup? How often? 

And this happens all of a sudden? It has been no problem till now?

Respectfully,

Z_A_P_a0ce3f

1 Rookie

 • 

3 Posts

November 24th, 2023 11:28

@DELL-Young E​ 

Please see below for the list of vols, snapshots and corresponding tasks.
When currently trying to snap a 5tb vol, the operation fails with error "... the total snapshot space is too large". When snapping a 1tb vol, we can do so 4 times in a row but a 5th snap fails with the same error.
To us it very much looks like the total space for snapshots is somehow limited. Is it?
The only info we can find is this:
https://www.dell.com/support/manuals/en-us/powervault-me4024/me4_series_ag_pub/creating-snapshots?guid=guid-ef9f7a11-49db-46a0-a8b9-65837c61c791&lang=en-us
But the large-pools option is disabled on our system.
Is the any other snapshot (storage) limit?
Thanks

#  show volumes type base
Pool Name             Total Size Alloc Size Class    Type Large Virtual Extents  Health Reason Action
----------------------------------------------------------------------------------------------------------------
A    Vol0001          1024.0GiB  210.5GiB   Virtual  base Disabled               OK
A    Vol0002          100.0GiB   0iB        Virtual  base Disabled               OK
A    Vol0002-xxx      25.0TiB    7776.9GiB  Virtual  base Disabled               OK
A    Vol0002-user     20.0TiB    7948.4GiB  Virtual  base Disabled               OK
A    Vol0003          4096.0GiB  130.1GiB   Virtual  base Disabled               OK
A    Vol0004-xxx      1024.0GiB  376.9GiB   Virtual  base Disabled               OK
A    Vol0005-xxx      50.0TiB    39.3TiB    Virtual  base Disabled               OK
A    Vol0006-xxx      1024.0GiB  460.0MiB   Virtual  base Disabled               OK
A    Vol0007-xxx      5120.0GiB  1394.0GiB  Virtual  base Disabled               OK
A    Vol0008-xxx      1024.0GiB  112.0MiB   Virtual  base Disabled               OK
A    Vol0009-xxx      5120.0GiB  1223.9GiB  Virtual  base Disabled               OK
A    Vol0010-xxx      50.0TiB    5801.1GiB  Virtual  base Disabled               OK
A    Vol0011-xxx      25.0TiB    278.2GiB   Virtual  base Disabled               OK
A    Vol0020-xxx      8192.0GiB  0iB        Virtual  base Disabled               OK
A    Vol0021-xxx      16.0TiB    46.2GiB    Virtual  base Disabled               OK
A    Vol_tmp_01       100.0GiB   9.2GiB     Virtual  base Disabled               OK
A    Vol_tmp_02       100.0GiB   24.2GiB    Virtual  base Disabled               OK
A    Vol_tmp_04       10.0GiB    0iB        Virtual  base Disabled               OK
A    vol_tmp_03       100.0GiB   41.4GiB    Virtual  base Disabled               OK
----------------------------------------------------------------------------------------------------------------

# show schedules
Name              Specification                                                      Status Next Time              Task To Run             Error Message
---------------------------------------------------------------------------------------------------------------------------------------------------------
Vol0001SS001      Start 2020-06-05 01:00:00, Every 1 Days                            Ready  2023-11-25 01:00:00    Vol0001TS001
Vol0001SS002      Start 2020-09-27 02:00:00, Every 1 Days, Only Any Sunday Of Month  Ready  2023-11-26 02:00:00    Vol0001TS002
Vol0001SS003      Start 2021-05-01 02:30:00, Every 1 Days, Only 1st Day Of Month     Ready  2023-12-01 02:30:00    Vol0001TS003
Vol0002-xxxSS001  Start 2023-07-13 01:30:00, Every 1 Days                            Ready  2023-11-25 01:30:00    Vol0002-xxxTS001
Vol0002-xxxSS002  Start 2023-07-16 02:00:00, Every 1 Days, Only Any Sunday Of Month  Ready  2023-11-26 02:00:00    Vol0002-xxxTS002
Vol0002-xxxSS003  Start 2023-08-01 02:30:00, Every 1 Days, Only 1st Day Of Month     Ready  2023-12-01 02:30:00    Vol0002-xxxTS003
Vol0002-userSS001 Start 2023-06-07 01:00:00, Every 1 Days                            Ready  2023-11-25 01:00:00    Vol0002-userTS001
Vol0002-userSS002 Start 2023-06-11 02:00:00, Every 1 Days, Only Any Sunday Of Year   Ready  2023-11-26 02:00:00    Vol0002-userTS002
Vol0002-userSS003 Start 2023-07-01 02:30:00, Every 1 Days, Only 1st Day Of Month     Ready  2023-12-01 02:30:00    Vol0002-userTS003
Vol0003SS001      Start 2021-04-11 01:00:00, Every 1 Days                            Ready  2023-11-25 01:00:00    Vol0003TS001
Vol0003SS002      Start 2021-04-11 02:00:00, Every 1 Days, Only Any Sunday Of Year   Ready  2023-11-26 02:00:00    Vol0003TS002
Vol0004-xxxSS001  Start 2020-08-17 23:30:00, Every 1 Days                            Ready  2023-11-24 23:30:00    Vol0004-xxxTS001
Vol0005-xxxSS001  Start 2020-08-20 02:30:00, Every 1 Days                            Ready  2023-11-25 02:30:00    Vol0005-xxxTS001
Vol0007-xxxSS001  Start 2020-08-20 02:30:00, Every 1 Days                            Ready  2023-11-25 02:30:00    Vol0007-xxxTS001
Vol0008-xxxSS001  Start 2023-08-30 02:30:00, Every 1 Days                            Ready  2023-11-25 02:30:00    Vol0008-xxxTS001
Vol0009-xxxSS001  Start 2021-11-30 02:30:00, Every 1 Days                            Ready  2023-11-25 02:30:00    Vol0009-xxxTS001
Vol0010-xxxSS001  Start 2021-11-30 02:30:00, Every 1 Days                            Ready  2023-11-25 02:30:00    Vol0010-xxxTS001
Vol0011-xxxSS001  Start 2023-08-30 02:30:00, Every 1 Days                            Ready  2023-11-25 02:30:00    Vol0011-xxxTS001
---------------------------------------------------------------------------------------------------------------------------------------------------------

# show tasks
Name              Type          Status State       Error Message
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Vol0001TS001      TakeSnapshot  Ready  Start
Vol0001TS002      TakeSnapshot  Ready  Start
Vol0001TS003      TakeSnapshot  Ready  Start
Vol0002xxxTS001   TakeSnapshot  Ready  Start
Vol0002xxxTS002   TakeSnapshot  Error  CreateSnap  Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0002xxxTS003   TakeSnapshot  Error  CreateSnap  Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0002-userTS001 TakeSnapshot  Error  CreateSnap  Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0002-userTS002 TakeSnapshot  Error  CreateSnap  Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0002-userTS003 TakeSnapshot  Error  CreateSnap  Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0003TS001      TakeSnapshot  Ready  Start
Vol0003TS002      TakeSnapshot  Ready  Start
Vol0004-xxxTS001  TakeSnapshot  Ready  Start
Vol0005-xxxTS001  TakeSnapshot  Ready  Start
Vol0007-xxxTS001  TakeSnapshot  Ready  Start
Vol0008-xxxTS001  TakeSnapshot  Ready  Start
Vol0009-xxxTS001  TakeSnapshot  Ready  Start
Vol0010-xxxTS001  TakeSnapshot  Error  CreateSnap  Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0011-xxxTS001  TakeSnapshot  Error  CreateSnap  Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

DELL-Erman O

Moderator

 • 

2.3K Posts

24-11-2023 12:50 PM

Hi, I've read this article. PowerVault ME4: What is Overcommitment and how does it work? | Dell Dominica The ME4 system can only store snapshots up to half of its total space. If this limit is reached, no new snapshots can be created and an error will show up. Just a brainstorming to avoid the snapshot space limit, you can either delete old snapshots, schedule fewer snapshots, or enable a larger pool option.(which will increase the limit to 80% of the total capacity of the system)

 

DELL-Erman O

Social Media and Communities Professional

Dell Technologies | Enterprise Support Services

#IWork4Dell

Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

Z_A_P_a0ce3f

1 Rookie

 • 

3 Posts

November 24th, 2023 15:00

@Dell-ErmanO​ 
Thanks for your answer and the link. It's an interesting reading and brings some light into the storage architecture of the ME4.
Although I can't find any information about a 50% snapshot limit in the document, I don't think that's our problem. Looks like our current snapshot size is way smaller.

# show pools
Name Class    Blocksize Total Size Avail    Snap Size OverCommit  Disk Groups Volumes  Low Thresh  Mid Thresh  High Thresh  Sec Fmt   Health
---------------------------------------------------------------------------------------------------------------------------------------------
A    Virtual  512       258.4TiB   192.2TiB 2278.5GiB Enabled     2           114      50.00 %     75.00 %     99.92 %      512e      OK
---------------------------------------------------------------------------------------------------------------------------------------------

But what about "... the FW supports a max of 1 PiB of Metadata Index."? Is it possible that we are hitting the metadata limit of the system?

DELL-Erman O

Moderator

 • 

2.3K Posts

24-11-2023 15:28 PM

hmm when you mentioned it, I thought about whether this could be metadata. But I'm not sure, we might be missing something. Now if so you think you're reaching the metadata limit, review and reduce your volumes and snapshots if possible to free up metadata space might be helpful. However let's see how our community contributes.

DELL-Erman O

Social Media and Communities Professional

Dell Technologies | Enterprise Support Services

#IWork4Dell

Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

View More

View All

No Events found!

Top