SRDF problem

interesting ..why is post from April 8 would show up at the very top of the list even though there are posts from a couple of minutes ago ...is somebody modifying it ?

Hmmm... That's not good. Your R2 thinks it has 16 invalid tracks on itself:

       Source (R1) View                 Target (R2) View     MODES
--------------------------------    ------------------------ ----- ------------
             ST                  LI      ST
Standard      A                   N       A
Logical       T  R1 Inv   R2 Inv  K       T  R1 Inv   R2 Inv       RDF Pair
Device  Dev   E  Tracks   Tracks  S Dev   E  Tracks   Tracks MDA   STATE
-------------------------------- -- ------------------------ ----- ------------
 
DEV001  0B01 WD       0        1 NR 0A9F RW       0       16 S..   Failed Over

...which is why it will not allow you to fail back. The R1 however looks clean, so if you've not made any significant changes to the R2 whilst failed over, I would do an srdf establish (possibly with a "-force") to cause those 16 tracks to be overwritten by the "good" copy from the R1 side. There is also 1 R1 track owed to the R2 (which in itself is unusual after a failover, since the R1 is write disabled).

Without detailed logs its not possible to say how it got in this state. Open an SR with EMC if you need further investigation.

Best Regards,
Marc

Thanks Mark and I tried to establish on R1/R2, failed:


R1:
C:\Documents and Settings\Administrator>symrdf -g tgtdg establish -force

Execute an RDF 'Incremental Establish' operation for device
group 'tgtdg' (y/ ) ? y

An RDF 'Incremental Establish' operation execution is
in progress for device group 'tgtdg'. Please wait...


The device is not in a valid RDF state for this operation






R2:
C:\Documents and Settings\Administrator>symrdf -g srcdg establish -force

Execute an RDF 'Incremental Establish' operation for device
group 'srcdg' (y/ ) ? y

An RDF 'Incremental Establish' operation execution is
in progress for device group 'srcdg'. Please wait...


The device is not in a valid RDF state for this operation


C:\Documents and Settings\Administrator>

Do a "split -force" to separate the pair first, then a normal establish will restart the sync.

still failed:
R1:
C:\Documents and Settings\Administrator>symrdf -g srcdg split -force

Execute an RDF 'Split' operation for device
group 'srcdg' (y/ ) ? y

An RDF 'Split' operation execution is
in progress for device group 'srcdg'. Please wait...


The device is not in a valid RDF state for this operation


C:\Documents and Settings\Administrator>symrdf -g srcdg split -force -symforce

Execute an RDF 'Split' operation for device
group 'srcdg' (y/ ) ? y

An RDF 'Split' operation execution is
in progress for device group 'srcdg'. Please wait...


The device is not in a valid RDF state for this operation


C:\Documents and Settings\Administrator>



R2:
C:\Documents and Settings\Administrator>symrdf -g tgtdg split -force

Execute an RDF 'Split' operation for device
group 'tgtdg' (y/ ) ? y

An RDF 'Split' operation execution is
in progress for device group 'tgtdg'. Please wait...


The device is not in a valid RDF state for this operation


C:\Documents and Settings\Administrator>

You should be able to do the following:

symrdf -g srcdg rw_enable r2 -force


This will put the group into a SPLIT state. From here, you must decide which data to keep. If you want to have R1 overwrite the R2, issue:

symrdf -g srcdg establish

If you want to have the R2 overwrite the R1, issue:

symrdf -g srcdg restore

Thank you very much John and the problem was fixed. But the R2 is still thinking 16 tracks invalide although R1 is clean. Actually there is no update on R1/R2 and I am just doing the test.

R1:
C:\Documents and Settings\Administrator>symrdf -g srcdg rw_enable r1 -force

Execute an RDF 'Read/Write Enable R1' operation for device
group 'srcdg' (y/ ) ? y

An RDF 'Read/Write Enable R1' operation execution is
in progress for device group 'srcdg'. Please wait...

Read/Write Enable device(s) on SA at source (R1)..........Done.

The RDF 'Read/Write Enable R1' operation successfully executed for
device group 'srcdg'.


C:\Documents and Settings\Administrator>symrdf -g srcdg query


Device Group (DG) Name : srcdg
DG's Type : RDF1
DG's Symmetrix ID : 000185xxxxx0




Source (R1) View Target (R2) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0B01 RW 0 1 NR 0A9F RW 0 16 S.. Split

Total -------- -------- -------- --------
MB(s) 0.0 0.0 0.0 0.5

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


C:\Documents and Settings\Administrator>



R2:
C:\Documents and Settings\Administrator>symrdf -g tgtdg query


Device Group (DG) Name : tgtdg
DG's Type : RDF2
DG's Symmetrix ID : 000185xxxxx8




Target (R2) View Source (R1) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0A9F RW 0 16 NR 0B01 RW 0 1 S.. Split

Total -------- -------- -------- --------
MB(s) 0.0 0.5 0.0 0.0

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


C:\Documents and Settings\Administrator>







Establish:

R1:
C:\Documents and Settings\Administrator>symrdf -g srcdg establish

Execute an RDF 'Incremental Establish' operation for device
group 'srcdg' (y/ ) ? y

An RDF 'Incremental Establish' operation execution is
in progress for device group 'srcdg'. Please wait...

Write Disable device(s) on RA at target (R2)..............Done.
Suspend RDF link(s).......................................Done.
Merge device track tables between source and target.......Started.
Devices: 0B01-0B08 ...................................... Merged.
Merge device track tables between source and target.......Done.
Resume RDF link(s)........................................Done.

The RDF 'Incremental Establish' operation successfully initiated for
device group 'srcdg'.


C:\Documents and Settings\Administrator>symrdf -g srcdg query


Device Group (DG) Name : srcdg
DG's Type : RDF1
DG's Symmetrix ID : 000185xxxxx0




Source (R1) View Target (R2) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0B01 RW 0 6 RW 0A9F WD 0 16 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 0.2 0.0 0.5

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


C:\Documents and Settings\Administrator>



R2:
C:\Documents and Settings\Administrator>symrdf -g tgtdg query


Device Group (DG) Name : tgtdg
DG's Type : RDF2
DG's Symmetrix ID : 000185xxxxx8




Target (R2) View Source (R1) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0A9F WD 0 16 RW 0B01 RW 0 6 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 0.5 0.0 0.2

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


C:\Documents and Settings\Administrator>



10 minutes:

R1:
C:\Documents and Settings\Administrator>symrdf -g srcdg query


Device Group (DG) Name : srcdg
DG's Type : RDF1
DG's Symmetrix ID : 000185xxxxx0




Source (R1) View Target (R2) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0B01 RW 0 0 RW 0A9F WD 0 16 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 0.0 0.0 0.5

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


C:\Documents and Settings\Administrator>

C:\Documents and Settings\Administrator>symrdf -g tgtdg query


Device Group (DG) Name : tgtdg
DG's Type : RDF2
DG's Symmetrix ID : 000185xxxxx8




Target (R2) View Source (R1) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0A9F WD 0 16 RW 0B01 RW 0 0 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 0.5 0.0 0.0

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off

what about doing a full establish with the only device effected(R1 to R2) . Not the entire devices from symdg but file based.

It should eventually come back in a synchronized state. If it doesn't come back within a 1/2 hour, I'd suggest opening a call with EMC support to have the PSE check out the R2 device.

John, I tried a few tests, always on R2 side, differnt disks, the R2 side had some invalid tracks. These 2 Symm 8830 are decomm and EMC does not want to support it anymore. I am just playing with it and no prodution on it

After I forced to split, now starting a full establish, seems like the R2 side still not working well, it still sees 16 invalid track.

R1:
C:\Documents and Settings\Administrator>symrdf -g srcdg establish -full

Execute an RDF 'Full Establish' operation for device
group 'srcdg' (y/ ) ? y

An RDF 'Full Establish' operation execution is
in progress for device group 'srcdg'. Please wait...

Write Disable device(s) on RA at target (R2)..............Done.
Suspend RDF link(s).......................................Done.
Mark target (R2) devices for full copy from source (R1)...Started.
Device: 0B01 ............................................ Marked.
Device: 0B02 ............................................ Marked.
Device: 0B03 ............................................ Marked.
Device: 0B04 ............................................ Marked.
Device: 0B05 ............................................ Marked.
Device: 0B06 ............................................ Marked.
Device: 0B07 ............................................ Marked.
Device: 0B08 ............................................ Marked.
Mark target (R2) devices for full copy from source (R1)...Done.
Merge device track tables between source and target.......Started.
Devices: 0B01-0B08 ...................................... Merged.
Merge device track tables between source and target.......Done.
Resume RDF link(s)........................................Done.

The RDF 'Full Establish' operation successfully initiated for
device group 'srcdg'.


C:\Documents and Settings\Administrator>symrdf -g srcdg query


Device Group (DG) Name : srcdg
DG's Type : RDF1
DG's Symmetrix ID : 000185xxxxx0




Source (R1) View Target (R2) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0B01 RW 0 1076349 RW 0A9F WD 0 16 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 33635.9 0.0 0.5

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


C:\Documents and Settings\Administrator>

R2:
C:\Documents and Settings\Administrator>symrdf -g tgtdg query


Device Group (DG) Name : tgtdg
DG's Type : RDF2
DG's Symmetrix ID : 000185xxxxx8




Target (R2) View Source (R1) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0A9F WD 0 16 RW 0B01 RW 0 1034589 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 0.5 0.0 32330.9

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off

After a few hours, the R2 still has 16 invalid tracks although there is no any update on both R1/R2:

R1:
C:\Documents and Settings\Administrator>symrdf -g srcdg query


Device Group (DG) Name : srcdg
DG's Type : RDF1
DG's Symmetrix ID : 000185xxxxx0




Source (R1) View Target (R2) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0B01 RW 0 0 RW 0A9F WD 0 16 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 0.0 0.0 0.5

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


R2:
C:\Documents and Settings\Administrator>symrdf -g tgtdg query


Device Group (DG) Name : tgtdg
DG's Type : RDF2
DG's Symmetrix ID : 000185xxxxx8




Target (R2) View Source (R1) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0A9F WD 0 16 RW 0B01 RW 0 0 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 0.5 0.0 0.0

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off





Here was how I got to my original status-->failover:

R1:
C:\Documents and Settings\Administrator>symrdf -g srcdg query


Device Group (DG) Name : srcdg
DG's Type : RDF1
DG's Symmetrix ID : 000185xxxxx0




Source (R1) View Target (R2) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0B01 RW 0 0 RW 0A9F WD 0 16 S.. SyncInProg

Total -------- -------- -------- --------
MB(s) 0.0 0.0 0.0 0.5

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


C:\Documents and Settings\Administrator>symrdf -g srcdg failover

Execute an RDF 'Failover' operation for device
group 'srcdg' (y/ ) ? y

An RDF 'Failover' operation execution is
in progress for device group 'srcdg'. Please wait...


Cannot proceed in the current RDF state except if the Symmetrix force flag is us
ed. Note that the Symmetrix force flag is not allowed to be used by default in R
DF control operations


C:\Documents and Settings\Administrator>symrdf -g srcdg failover -symforce

Execute an RDF 'Failover' operation for device
group 'srcdg' (y/ ) ? y

An RDF 'Failover' operation execution is
in progress for device group 'srcdg'. Please wait...

Write Disable device(s) on SA at source (R1)..............Done.
Suspend RDF link(s).......................................Done.
Read/Write Enable device(s) on RA at target (R2)..........Done.

The RDF 'Failover' operation successfully executed for
device group 'srcdg'.


C:\Documents and Settings\Administrator>

C:\Documents and Settings\Administrator>symrdf -g srcdg query


Device Group (DG) Name : srcdg
DG's Type : RDF1
DG's Symmetrix ID : 000185xxxxx0




Source (R1) View Target (R2) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0B01 WD 0 0 NR 0A9F RW 0 16 S.. Failed Over

Total -------- -------- -------- --------
MB(s) 0.0 0.0 0.0 0.5

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off



R2:
C:\Documents and Settings\Administrator>symrdf -g tgtdg query


Device Group (DG) Name : tgtdg
DG's Type : RDF2
DG's Symmetrix ID : 000185xxxxx8




Target (R2) View Source (R1) View MODES
-------------------------------- ------------------------ ----- ------------
ST LI ST
Standard A N A
Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair
Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0A9F RW 0 16 NR 0B01 WD 0 0 S.. Failed Over

Total -------- -------- -------- --------
MB(s) 0.0 0.5 0.0 0.0

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino) : X = Enabled, . = Disabled
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off

It's RAID 10.

C:\Documents and Settings\Administrator>symcli


Symmetrix Command Line Interface (SYMCLI) Version V5.5.0.0 (Edit Level: 557)
built with SYMAPI Version V5.5.0.0 (Edit Level: 557)

Are these RAID5 devices?

Also, what version of Solutions Enabler are you using?

Honestly, it would probably require someone dialing into the box to fix this.

You might try rebuilding the RDF pair to see if that resolves the issue.