Why symclone vse terminate takes very long time?

Hi Harry,

You need to open an SR with the Support Center for this.  When a a VP snap session is terminated, there is a "terminate-deallocate" task that is run to deallocate the data from the target and/or split shared allocations from shared group.  This can take "time" but the times you list seem far too long.  If the source only has a single target then the VSE won't have any shared allocations, this is something the support center will need to know.

Keep in mind that even if the customer is not writing directly to the target(s), the target(s) may have allocations/shared allocations because tracks will be copied from source to target during source write.  So even though the customer is only reading from the targets and not writing to them, the targets may not be empty.

Also, I would not suggest using -force or -symforce unless you are doing so for a specific reason.  Does the terminate take just as long without the -symforce?  Makes me wonder if it would be rejected if not forced, but instead it's being forced through and this has to do with the long terminate.  Just a guess on that, the support center may want to know this as well.

Good luck, please update the discussion with the findings!

Mike

Hi Micheal,

Thanks for prompt reply !

I think need to explain more about the scenario. It is a cascaded CloneVP--VPsnap configuration of 9 data drives. In the order to provide the database for reporting purpose to end user we have below process steps:

1. Clone VP running in precopy state.
2. Flush the cache and unmount VPsnap targets.
3. Terminate the VPsnap session.
4. Activate the Clone VP to full copy.
5. Wait till all reach to copied state.
6. Establish the cascaded VPsnap.
7. Mount the VPsnap target back for end user.
8. Recreate the Clone VP in precopy state for next cycle.

• There is no multi-allocation of any pair.
• No writes seen in target devices.

Symforce option is EMC suggested only. And I agreed with terminate deallocate process, but it should finish quickly, if there is no modified track @ target devices.

Questions are why Terminate deallocate takes long time? Why problem is intermittent, if there are negligible changes in respect of source data size? What i can do to boost this process?

Hi Harry

As for an exact reason why the terminate is taking long intermittently, you will need to contact Support for help with that. They may want to check the status of the devices/sessions before terminate, and then observe the terminate as it happens. I believe when you say they have no allocated data but basic troubleshooting will require to verify this with Inlines before starting the terminate.

But I have a few comments on the process you outlined.

I am trying to understand why you don't just terminate the VP Snap session before recreating the clone session. Is there a reason? Since you are terminating the VP snap session anyway, it would be better to terminate first and then do your recreate (with or without precopy) and active of clone, then create/activate the cascaded VP snaps.

Another option would be to never terminate the Cascaded vp snaps, just issue recreate/activate to them as needed. But every time you recreate the clone with the cascaded vp snap intact there is the possibility of copying data to the vp snap which may increase backend utilization. Terminating will avoid that.

The precopy on the clone will copy tracks to the vp snap targets (depending on how much source or intermediate clone was updated). If you use the suggested steps below and create /activate vp snaps, then mount and read from them, and then terminate them before recreating the clone session, there should be no data to deallocate and the terminate should be quick.

I stand by the recommendation to NOT use -force or -symforce. Are you saying someone from EMC told you to use them every time in the steps above? Are you using it at step 3, or anywhere else? If you are not allowed to do something without it then it is indicative of a procedure issue and need to find the proper way to proceed so that a force is not needed. Im thinking maybe step 3 needs force because the clone is not in an active state. If you use the flow I suggested you should not need to force anything through.

Let me know if you can try the different order of steps and if it helps! It would look something like this:

1. Recreate and activate clone session (precopy optional)

2. Wait for copied state

3. Establish cascaded vp snap

4. Mount vp snap

5. Flush cache and unmount vp snap

6. Terminate vp snap

7. Back to step 1

Mike

Sent from my iPhone

Hi Mike,

Thanks for showing interest !

As i mentioned, i have plenty of data. And have a very little time window (40 minutes) to complete the whole process from storage point of view. If i will go with your steps

1. Recreate and activate clone session (precopy optional)  (At Database level whole day changes can be huge )

2. Wait for copied state                 (Due to enough changes, diff. session will take long time to reach copied state) 

3. Establish cascaded vp snap                  

4. Mount vp snap

5. Flush cache and unmount vp snap          (In my scenario, this is 1st step of BCV start)

6. Terminate vp snap

7. Back to step 1

Now why we have to stick to below process steps, because:

1. Clone VP running in precopy state.      (It precopy the required data throughout the day)
2. Flush the cache and unmount VPsnap targets.    (1st step of our 40 minutes process)
3. Terminate the VPsnap session.                 (As per EMC WP Recreate is nothing but Terminate-Create New)
4. Activate the Clone VP to full copy.             (as per business, we need to activate the clone in b/w 40 mins)
5. Wait till all reach to copied state.                (it completes fast on expectation because begun with precopy)
6. Establish the cascaded VPsnap.              
7. Mount the VPsnap target back for end user.
8. Recreate the Clone VP in precopy state for next cycle.   (last step, starts precopy session for next day cycle)

Yes, symforce is @ 3 to terminate the VPSnap session only. symforce is just an option to avoid any kind of hitch, not mendatory. Earlier it wasn't there, but we used it for safer side after EMC suggestion. I think it is not considerable point in this problem.

Hi Harry,

Thanks for the additional info, helps understanding for sure. I thought you were using precopy only because you have a cascaded VP Snap (in which case it is required). If you try the other order of procedure, the commands will not require it, but you certainly can use it, and sounds like it might be a good idea for your environment.

I checked the SE TF guide and terminate of the Clone is allowed with the cascaded VP Snap active. I couldn’t get to the guide yesterday and thought maybe the symforce was required here, but it is not. I would still recommend to not use it unless necessary and for a specific reason. But like you say, I agree this is probably not causing any problem right now.

What I think is happening is that since you are keeping the clone in precopy, and sounds like many changes/writes to the source, so data is propagating from the intermediate clone to the VP Snap, and they need to be deallocated at terminate. So you probably need to make a decision if it is more important to keep your order of steps and accept the long terminate times, or if terminate time is more important, then reorder the steps as I suggested. One thing I left out, since you want to continue to use precopy, would be to verify that precopy as cycled before activating.

1. Recreate Clone session with precopy

2. Verify precopy has cycled

3. Activate clone session

4. Wait for copied state

5. Establish cascaded vp snap

6. Mount vp snap

7. Flush cache and unmount vp snap

8. Terminate vp snap

9. Back to step 1

If you want I can dial into the VMAX and see how many tracks are allocated on the VP Snaps before terminate (send me the serial number offline). From there, if you still want a deeper analysis of the exact terminate time, you would need to contact Support, they may want to see the terminate in progress, set traces, have someone from performance look, etc.

Mike

Hi Mike,

Thanks for another prompt reponse and taking it more interestingly !

I forgot to mention, verification of precopy cycled state, which is already in place. Your suggested steps are exactly similar to what we have, but difference of loop start step, that doesn't matter much. 

I have already placed SR for it. But to analyze the casue of intermittently facing termination delay, please suggest what i can check further?

Hi Harry,

There isn’t much else to check from this point. It pretty much comes down to how much data needs to be deallocated from the VP Snap when it is terminated, and then if it is being deallocated quickly enough, which is what Support will need to check. The only thing to try would be the re-order of the steps.

There is a difference in your steps and what I outlined. You have the VP Snap active while the clone is in precopy. This is likely causing data to be copied to the VP Snaps, which then needs to be deallocated at terminate. The steps I describe create the VP Snap after the clone has been activated, and then terminate the VP Snap before the clone is recreated with precopy. So nothing will be allocated on the VP Snaps and terminate/deallocate should complete quicker.

Can you please update the discussion with the findings from the SR? Would be interesting to know if Support finds a bug or another issue, or not.

Mike

• Clone VP running in precopy cycled state.     
• Flush the cache and unmount VPsnap targets. 
• Terminate the VPsnap session.               
• Activate the Clone VP to full copy.          
• Wait till all reach to copied state.               
• Establish and activate the cascaded VPsnap.              
• Mount the VPsnap target back for end user.
• Recreate the Clone VP in precopy state for next cycle.

I dont think so. VP snap has activated before clone recreate.

Sure, I will update the SR findings.

Hi Harry,

I’m not sure what you mean by “I don’t think so.”  Can you explain?

The order you are running the steps, you will not see any data copied to the VP Snaps when you first activate them.  But, data will copy to them when you recreate (with precopy) the Clone session.  This will obviously be dependent on how many updates need to copy, but precopy will send data from source to Clone, which will then in turn copy data from Clone to VP Snap, same as if you were to write directly to the Clone (again, dependent on how many updates need to copy).  And this will affect terminate time.

If you want to completely avoid copying any data to the VP Snap, then the steps should be done in this order:

·        1. Recreate Clone session with precopy

·        2.    Verify precopy has cycled

·        3.    Activate clone session

·        4.    Wait for copied state

·        5.    Establish cascaded vp snap

·        6.    Mount vp snap

·        7.    Flush cache and unmount vp snap

·        8.    Terminate vp snap

·        9.    Back to step 1

Before terminating, you can check how much data is allocated on the VP Snaps with command symcfg –sid list –tdev –devs

Mike

The difference in the procedures is the effect on the VP Snap targets. You need to trust me on this ☺

But I understand that your business requirements may prevent you from changing procedures. I cannot say for sure that your other steps can be done in the required time. The only way I can think of is to try it, if possible. If you cannot change procedures, then the only thing to do is let Support verify that the code is behaving properly.

Either way, try that symcfg command to see how much is allocated on the VP Snap before terminate.

Good luck!

Mike