Unsolved
This post is more than 5 years old
49 Posts
0
623
April 6th, 2009 14:00
RM BCV job not making devices ready
I have created a new BCV job, same setup as all others I have, but for some reason this one job is unable to mount the replicate once the consistent split is complete. I was able to trace it back to the 2 devices used in the replicate not being set to "ready" once the split is complete. I have other jobs on this server setup the same way and they run fine, but this one job is having trouble. If I clear the locks, I can manually set the devices to the ready state and mount them, but for some reason RM is not doing this for me. Has anyone see this before? Client is Solaris 10 with SE 6.2.5, RM server is 5.2 running on Windows 2003 server. Below is the symapi log when I try to mount the replica, it looks like the "ready succeeds and then for what ever reason 3 seconds later it makes them "not_ready"
Thanks in advance for any help...
04/06/2009 14:48:23.129 26953 STARTING a Dev Lock Release operation. [lock id: 9, device id: 4c1]
04/06/2009 14:48:23.142 26953 The Dev Lock Release operation SUCCEEDED.
04/06/2009 14:48:23.145 26953 STARTING a Dev Lock Release operation. [lock id: 9, device id: 4c7]
04/06/2009 14:48:23.200 26953 The Dev Lock Release operation SUCCEEDED.
04/06/2009 14:48:53.761 26953 STARTING a 'READY' control operation.
04/06/2009 14:48:53.761 26953 129 EMC:ReplicationMgr SymDevControl() symid: $SYM_ID symdev: 4C1 director: N/A, port: N/A ()
04/06/2009 14:48:53.791 26953 129 EMC:ReplicationMgr test_dev_state() TestState: DEV_STATE_READY_DEVICE, SymID: $SYM_ID Dev: (04C1) Remote: 0, Src: 0, BCV: 0, cfg: Inv Dev sts: NR, SA: RW, RA: RW, DA: RW, Link: RW, Consistency: DIS, Susp Pend: 0, BCV: 1024, BCV Paired: 1 BCV State: Split, InvR1: 0, InvR2: 0, R2WPonR1: 0 Mode: SYN, Dom: ENB, ACp: OFF, API sts: SYMAPI_C_INV_DEVICE_RDY_STATUS
04/06/2009 14:48:53.826 26953 Set device(s) Ready at local Symmetrix....................Done.
04/06/2009 14:48:53.835 26953 The 'READY' control operation SUCCEEDED.
04/06/2009 14:48:54.046 26953 STARTING a 'READY' control operation.
04/06/2009 14:48:54.046 26953 129 EMC:ReplicationMgr SymDevControl() symid: $SYM_ID symdev: 04C7 director: N/A, port: N/A ()
04/06/2009 14:48:54.068 26953 129 EMC:ReplicationMgr test_dev_state() TestState: DEV_STATE_READY_DEVICE, SymID: $SYM_ID Dev: (04C7) Remote: 0, Src: 0, BCV: 0, cfg: Inv Dev sts: NR, SA: RW, RA: RW, DA: RW, Link: RW, Consistency: DIS, Susp Pend: 0, BCV: 1024, BCV Paired: 1 BCV State: Split, InvR1: 0, InvR2: 0, R2WPonR1: 0 Mode: SYN, Dom: ENB, ACp: OFF, API sts: SYMAPI_C_INV_DEVICE_RDY_STATUS
04/06/2009 14:48:54.104 26953 Set device(s) Ready at local Symmetrix....................Done.
04/06/2009 14:48:54.114 26953 The 'READY' control operation SUCCEEDED.
04/06/2009 14:48:57.446 26953 STARTING a 'NOT_READY' control operation.
04/06/2009 14:48:57.446 26953 129 EMC:ReplicationMgr SymDevControl() symid: $SYM_ID symdev: 4C1 director: N/A, port: N/A ()
04/06/2009 14:48:57.467 26953 129 EMC:ReplicationMgr test_dev_state() TestState: DEV_STATE_NOT_READY_DEVICE, SymID: $SYM_ID Dev: (04C1) Remote: 0, Src: 0, BCV: 0, cfg: Inv Dev sts: RW, SA: RW, RA: RW, DA: RW, Link: RW, Consistency: DIS, Susp Pend: 0, BCV: 1024, BCV Paired: 1 BCV State: Split, InvR1: 0, InvR2: 0, R2WPonR1: 0 Mode: SYN, Dom: ENB, ACp: OFF, API sts: SYMAPI_C_INV_DEVICE_RDY_STATUS
04/06/2009 14:48:57.500 26953 Set device(s) Not Ready at local Symmetrix.................Done.
04/06/2009 14:48:57.508 26953 The 'NOT_READY' control operation SUCCEEDED.
04/06/2009 14:48:57.717 26953 STARTING a 'NOT_READY' control operation.
04/06/2009 14:48:57.717 26953 129 EMC:ReplicationMgr SymDevControl() symid: $SYM_ID symdev: 04C7 director: N/A, port: N/A ()
04/06/2009 14:48:57.748 26953 129 EMC:ReplicationMgr test_dev_state() TestState: DEV_STATE_NOT_READY_DEVICE, SymID: $SYM_ID Dev: (04C7) Remote: 0, Src: 0, BCV: 0, cfg: Inv Dev sts: RW, SA: RW, RA: RW, DA: RW, Link: RW, Consistency: DIS, Susp Pend: 0, BCV: 1024, BCV Paired: 1 BCV State: Split, InvR1: 0, InvR2: 0, R2WPonR1: 0 Mode: SYN, Dom: ENB, ACp: OFF, API sts: SYMAPI_C_INV_DEVICE_RDY_STATUS
04/06/2009 14:48:57.778 26953 Set device(s) Not Ready at local Symmetrix.................Done.
04/06/2009 14:48:57.786 26953 The 'NOT_READY' control operation SUCCEEDED.
04/06/2009 14:49:07.970 26953 STARTING a Dev Lock Acquire operation. [lock id: 9, device id: 4c1]
04/06/2009 14:49:07.987 26953 The Dev Lock Acquire operation SUCCEEDED.
04/06/2009 14:49:07.990 26953 STARTING a Dev Lock Acquire operation. [lock id: 9, device id: 4c7]
04/06/2009 14:49:08.020 26953 The Dev Lock Acquire operation SUCCEEDED.
Thanks in advance for any help...
04/06/2009 14:48:23.129 26953 STARTING a Dev Lock Release operation. [lock id: 9, device id: 4c1]
04/06/2009 14:48:23.142 26953 The Dev Lock Release operation SUCCEEDED.
04/06/2009 14:48:23.145 26953 STARTING a Dev Lock Release operation. [lock id: 9, device id: 4c7]
04/06/2009 14:48:23.200 26953 The Dev Lock Release operation SUCCEEDED.
04/06/2009 14:48:53.761 26953 STARTING a 'READY' control operation.
04/06/2009 14:48:53.761 26953 129 EMC:ReplicationMgr SymDevControl() symid: $SYM_ID symdev: 4C1 director: N/A, port: N/A ()
04/06/2009 14:48:53.791 26953 129 EMC:ReplicationMgr test_dev_state() TestState: DEV_STATE_READY_DEVICE, SymID: $SYM_ID Dev: (04C1) Remote: 0, Src: 0, BCV: 0, cfg: Inv Dev sts: NR, SA: RW, RA: RW, DA: RW, Link: RW, Consistency: DIS, Susp Pend: 0, BCV: 1024, BCV Paired: 1 BCV State: Split, InvR1: 0, InvR2: 0, R2WPonR1: 0 Mode: SYN, Dom: ENB, ACp: OFF, API sts: SYMAPI_C_INV_DEVICE_RDY_STATUS
04/06/2009 14:48:53.826 26953 Set device(s) Ready at local Symmetrix....................Done.
04/06/2009 14:48:53.835 26953 The 'READY' control operation SUCCEEDED.
04/06/2009 14:48:54.046 26953 STARTING a 'READY' control operation.
04/06/2009 14:48:54.046 26953 129 EMC:ReplicationMgr SymDevControl() symid: $SYM_ID symdev: 04C7 director: N/A, port: N/A ()
04/06/2009 14:48:54.068 26953 129 EMC:ReplicationMgr test_dev_state() TestState: DEV_STATE_READY_DEVICE, SymID: $SYM_ID Dev: (04C7) Remote: 0, Src: 0, BCV: 0, cfg: Inv Dev sts: NR, SA: RW, RA: RW, DA: RW, Link: RW, Consistency: DIS, Susp Pend: 0, BCV: 1024, BCV Paired: 1 BCV State: Split, InvR1: 0, InvR2: 0, R2WPonR1: 0 Mode: SYN, Dom: ENB, ACp: OFF, API sts: SYMAPI_C_INV_DEVICE_RDY_STATUS
04/06/2009 14:48:54.104 26953 Set device(s) Ready at local Symmetrix....................Done.
04/06/2009 14:48:54.114 26953 The 'READY' control operation SUCCEEDED.
04/06/2009 14:48:57.446 26953 STARTING a 'NOT_READY' control operation.
04/06/2009 14:48:57.446 26953 129 EMC:ReplicationMgr SymDevControl() symid: $SYM_ID symdev: 4C1 director: N/A, port: N/A ()
04/06/2009 14:48:57.467 26953 129 EMC:ReplicationMgr test_dev_state() TestState: DEV_STATE_NOT_READY_DEVICE, SymID: $SYM_ID Dev: (04C1) Remote: 0, Src: 0, BCV: 0, cfg: Inv Dev sts: RW, SA: RW, RA: RW, DA: RW, Link: RW, Consistency: DIS, Susp Pend: 0, BCV: 1024, BCV Paired: 1 BCV State: Split, InvR1: 0, InvR2: 0, R2WPonR1: 0 Mode: SYN, Dom: ENB, ACp: OFF, API sts: SYMAPI_C_INV_DEVICE_RDY_STATUS
04/06/2009 14:48:57.500 26953 Set device(s) Not Ready at local Symmetrix.................Done.
04/06/2009 14:48:57.508 26953 The 'NOT_READY' control operation SUCCEEDED.
04/06/2009 14:48:57.717 26953 STARTING a 'NOT_READY' control operation.
04/06/2009 14:48:57.717 26953 129 EMC:ReplicationMgr SymDevControl() symid: $SYM_ID symdev: 04C7 director: N/A, port: N/A ()
04/06/2009 14:48:57.748 26953 129 EMC:ReplicationMgr test_dev_state() TestState: DEV_STATE_NOT_READY_DEVICE, SymID: $SYM_ID Dev: (04C7) Remote: 0, Src: 0, BCV: 0, cfg: Inv Dev sts: RW, SA: RW, RA: RW, DA: RW, Link: RW, Consistency: DIS, Susp Pend: 0, BCV: 1024, BCV Paired: 1 BCV State: Split, InvR1: 0, InvR2: 0, R2WPonR1: 0 Mode: SYN, Dom: ENB, ACp: OFF, API sts: SYMAPI_C_INV_DEVICE_RDY_STATUS
04/06/2009 14:48:57.778 26953 Set device(s) Not Ready at local Symmetrix.................Done.
04/06/2009 14:48:57.786 26953 The 'NOT_READY' control operation SUCCEEDED.
04/06/2009 14:49:07.970 26953 STARTING a Dev Lock Acquire operation. [lock id: 9, device id: 4c1]
04/06/2009 14:49:07.987 26953 The Dev Lock Acquire operation SUCCEEDED.
04/06/2009 14:49:07.990 26953 STARTING a Dev Lock Acquire operation. [lock id: 9, device id: 4c7]
04/06/2009 14:49:08.020 26953 The Dev Lock Acquire operation SUCCEEDED.
No Events found!
desimus
49 Posts
0
April 6th, 2009 15:00
root@wwpxpsdb02 # symdev show 04C1 | grep "Device Status"
Device Status : Not Ready (NR)
BCV Device Status : Not Ready (NR BCV)
root@wwpxpsdb02 # symdev show 04C7 | grep "Device Status"
Device Status : Not Ready (NR)
BCV Device Status : Not Ready (NR BCV)
Here is a device on the same host in a separate pool which is working just fine and setup the exact same way. After the job run but before it mounts the meta device is made ready:
oot@wwpxpsdb02 # symdev show 04C8 | grep "Device Status"
Device Status : Ready (RW)
BCV Device Status : Ready (RW)
JamesBEMC
257 Posts
0
April 7th, 2009 00:00
The sequence of flow for RM when dealing with locks and ready/notready is
Device not included in RM Not locked
Device included in RM Locked (LDL Lock 9), Not Ready
Job starts Dev NR
Sync Dev NR (systematically anyways)
Split Dev NR
Mount RM sets Dev Ready
Unmount or mount failure RM sets Dev NR again
So, perhaps the NR is a red herring. Perhaps. What does the rm client debug log say as soon the mount starts to fail?
Cheers