Unsolved
This post is more than 5 years old
4 Posts
0
1357
January 29th, 2008 09:00
ERM isssues
Dear All,
We have the current setup:-
R1 --> server A
BCV--> server B
The syncing of R1-BCV is done through ERM software. The server A (runs ERM client), server B (runs ERM server, ERM client).
Since last week our syncing was working fine but since we had added the new R1 devices and associated the same with new BCV devices, the syncing is failing.
I am getting the following error while running job simulation:-
Prepare Discover
2008 01 29 12:09:24 Starting simulation of [application set: AS_XXX_FILESYSTEMS Description: Contains all filesystems for XXX Host: server100 Platform: SunOS 5.8 Filesystem:/oracle/xxx/data15 ....
....
....
Filesystem:/oracle/xxx/data6 / job: JOB_XXX_FILESYSTEMS Description: Replicate the XXX filesystems from serverA to serverB Rotation: 1 Mount Host: serverB Storage Criteria: SP_XXX_BACKUP Mount Host Command: NONE Alternate Mount Path: --Original Path-- Backup Host: NONE Backup Script: NONE Mount Options: Mount Meta Data Path:] at time Jan 29, 2008 12:09:24 PM.
2008 01 29 12:09:24 This operation can take a long time. Please be patient.
2008 01 29 12:09:31 000508 WARNING: Unable to find Navisphere CLI path. If CLARiiON arrays are being used, install Navisphere CLI into the default path: /opt/Navisphere/bin/. Navisphere CLI is not needed for Symmetrix arrays.
2008 01 29 12:09:31 000600 ERROR: Storage device /oracle/xxx/data13 could not be located on supported arrays. Please check if there are problems communicating with the storage arrays.
2008 01 29 12:09:31 000010 ERROR: An internal error occurred on the Replication Manager host as follows: processGetStorageDetails - failed to write storageDetails to ird.
2008 01 29 12:09:31 000024 ERROR: An unexpected internal error occurred: rawMessage::getSessionId - null buffer.
2008 01 29 12:09:31 000024 ERROR: An unexpected internal error occurred: rawMessage::getRequestId - null buffer.
2008 01 29 12:09:31 000508 WARNING: Unable to find Navisphere CLI path. If CLARiiON arrays are being used, install Navisphere CLI into the default path: /opt/Navisphere/bin/. Navisphere CLI is not needed for Symmetrix arrays.
2008 01 29 12:09:31 000600 ERROR: Storage device /oracle/xxx/data13 could not be located on supported arrays. Please check if there are problems communicating with the storage arrays.
2008 01 29 12:09:31 000010 ERROR: An internal error occurred on the Replication Manager host as follows: processGetStorageDetails - failed to write storageDetails to ird.
2008 01 29 12:09:31 000024 ERROR: An unexpected internal error occurred: rawMessage::getSessionId - null buffer.
2008 01 29 12:09:31 000024 ERROR: An unexpected internal error occurred: rawMessage::getRequestId - null buffer.
We have EMC symmetrix frame hence the Navi errors could be ignored, but what does the other errors indicate?
Also on serverB, I am unable to establish the R1-BCV syncing due to sym device lock error:-
# symmir -g devprd establish -full
Execute 'Full Establish' operation for device group
'sapprd' (y/ ) ? y
'Full Establish' operation execution is in progress for
device group 'devprd'. Please wait...
Unable to acquire the Symmetrix device lock
Please help...we are unable to take the backup at this point.
Also, ERM software does the following:-
serverB, serverA are sol 8 servers
so
1. it unmounts the EMC BCV devices and expires the replica.
2. it creates the new replica. sync and split.
3. enables the EMC BCV volume group in veritas
4. mounts it again.
Thanks,
Rahul.
We have the current setup:-
R1 --> server A
BCV--> server B
The syncing of R1-BCV is done through ERM software. The server A (runs ERM client), server B (runs ERM server, ERM client).
Since last week our syncing was working fine but since we had added the new R1 devices and associated the same with new BCV devices, the syncing is failing.
I am getting the following error while running job simulation:-
Prepare Discover
2008 01 29 12:09:24 Starting simulation of [application set: AS_XXX_FILESYSTEMS Description: Contains all filesystems for XXX Host: server100 Platform: SunOS 5.8 Filesystem:/oracle/xxx/data15 ....
....
....
Filesystem:/oracle/xxx/data6 / job: JOB_XXX_FILESYSTEMS Description: Replicate the XXX filesystems from serverA to serverB Rotation: 1 Mount Host: serverB Storage Criteria: SP_XXX_BACKUP Mount Host Command: NONE Alternate Mount Path: --Original Path-- Backup Host: NONE Backup Script: NONE Mount Options: Mount Meta Data Path:] at time Jan 29, 2008 12:09:24 PM.
2008 01 29 12:09:24 This operation can take a long time. Please be patient.
2008 01 29 12:09:31 000508 WARNING: Unable to find Navisphere CLI path. If CLARiiON arrays are being used, install Navisphere CLI into the default path: /opt/Navisphere/bin/. Navisphere CLI is not needed for Symmetrix arrays.
2008 01 29 12:09:31 000600 ERROR: Storage device /oracle/xxx/data13 could not be located on supported arrays. Please check if there are problems communicating with the storage arrays.
2008 01 29 12:09:31 000010 ERROR: An internal error occurred on the Replication Manager host as follows: processGetStorageDetails - failed to write storageDetails to ird.
2008 01 29 12:09:31 000024 ERROR: An unexpected internal error occurred: rawMessage::getSessionId - null buffer.
2008 01 29 12:09:31 000024 ERROR: An unexpected internal error occurred: rawMessage::getRequestId - null buffer.
2008 01 29 12:09:31 000508 WARNING: Unable to find Navisphere CLI path. If CLARiiON arrays are being used, install Navisphere CLI into the default path: /opt/Navisphere/bin/. Navisphere CLI is not needed for Symmetrix arrays.
2008 01 29 12:09:31 000600 ERROR: Storage device /oracle/xxx/data13 could not be located on supported arrays. Please check if there are problems communicating with the storage arrays.
2008 01 29 12:09:31 000010 ERROR: An internal error occurred on the Replication Manager host as follows: processGetStorageDetails - failed to write storageDetails to ird.
2008 01 29 12:09:31 000024 ERROR: An unexpected internal error occurred: rawMessage::getSessionId - null buffer.
2008 01 29 12:09:31 000024 ERROR: An unexpected internal error occurred: rawMessage::getRequestId - null buffer.
We have EMC symmetrix frame hence the Navi errors could be ignored, but what does the other errors indicate?
Also on serverB, I am unable to establish the R1-BCV syncing due to sym device lock error:-
# symmir -g devprd establish -full
Execute 'Full Establish' operation for device group
'sapprd' (y/ ) ? y
'Full Establish' operation execution is in progress for
device group 'devprd'. Please wait...
Unable to acquire the Symmetrix device lock
Please help...we are unable to take the backup at this point.
Also, ERM software does the following:-
serverB, serverA are sol 8 servers
so
1. it unmounts the EMC BCV devices and expires the replica.
2. it creates the new replica. sync and split.
3. enables the EMC BCV volume group in veritas
4. mounts it again.
Thanks,
Rahul.
0 events found
No Events found!
JamesBEMC
257 Posts
1
January 30th, 2008 04:00
What version of RM are you using?
The actual replication problem is noted by the error
2008 01 29 12:09:31 000600 ERROR: Storage device /oracle/xxx/data13 could not be located on supported arrays. Please check if there are problems communicating with the storage arrays.
RM has a problem resolving this filesystem to a Symm device. RM uses the SYMAPI version of the symrslv command
Run the following command on the production host
/usr/symcli/bin/symrslv fs /oracle/xxx/data13
Also the error - Unable to find Navisphere CLI path
You can ignore - this will be fixed in RM 5.1 Patch 1.
For - Unable to acquire the Symmetrix device lock
While the BCVs are under RM control (included storage), RM would have set Long duration locks (symm lock 9's) on those BCVs, so you will not be able to perform symmir commands against those devices.
2 choices
1) lift the lock 9's on those BCVs using symdev
or
2) remove the device temporarily from RM and then perform the same command.
I do recommend you open a support call in order to ascertain the errors returned from symrslv and for a deeper look into the rm client debug logs.
James.
rahulkhare
4 Posts
0
January 30th, 2008 13:00
Thanks for your reply. I had been banging my head with EMC tech support for two days now. I tried the symrslv command on the server where R1 is presented and it gave me the following error:-
# symrslv fs /oracle/xxx/data13
The attempt to interrogate a physical device associated with a volume group failed
However, the same command is working fine with the rootdg disk group (residing on local disk).
Thanks,
Rahul.
rahulkhare
4 Posts
0
January 31st, 2008 11:00
Your thread was the most accurate info I could have got for this issue. Thanks for pointing me to the correct direction. I did have a word with EMC and as per them I tried to go through the symrslv command issue. Here is the output of the same:-
iSymPrefsSet SYM_PREF_VERBOSE is being set to the value 1 (0X00000001)
SymPrefsSet Function Completed; status = 0 (The function call was successful) (handle: 0)
SymDataFileIdentify Function Enter (handle: 0)
emcCheckMapLicense Enter emcCheckMapLicense()
SymDataFileIdentify Enter SymDataFileIdentify(): dfname: /oracle/GEB/sapdata13
is_absolute_path_name Enter UNIX is_absolute_path_name(): data_file_name: /oracle/GEB/sapdata13
GetHostDefaultFsType Enter GetHostDefaultFsType()
check_map_license Enter check_map_license()
SymFileSystemGet Enter SymFileSystemGet(): real_fs_type: All file types
SunOSFileSystemGet Enter SunOSFileSystemGet()
SymDataFileIdentify Function Completed; status = 0 (The function call was successful) (handle: 0)
SymDataObjectResolve Function Enter (handle: 0)
emcCheckMapLicense Enter emcCheckMapLicense()
SymDataObjectResolve Enter SymDataObjectResolve(): type: File System objname: /oracle/GEB/sapdata13
emcCheckMapLicense Enter emcCheckMapLicense()
SymDataFileIdentify Enter SymDataFileIdentify(): dfname: /oracle/GEB/sapdata13
is_absolute_path_name Enter UNIX is_absolute_path_name(): data_file_name: /oracle/GEB/sapdata13
GetHostDefaultFsType Enter GetHostDefaultFsType()
check_map_license Enter check_map_license()
SymFileSystemGet Enter SymFileSystemGet(): real_fs_type: All file types
SunOSFileSystemGet Enter SunOSFileSystemGet()
cvt_fsys_to_resolve Enter cvt_fsys_to_resolve(): ip->abs_pathname: /oracle/GEB/sapdata13
emcCheckMapLicense Enter emcCheckMapLicense()
SymDataFileIdentify Enter SymDataFileIdentify(): dfname: /dev/vx/dsk/gebdg/oracle_GEB_sapdata13
is_absolute_path_name Enter UNIX is_absolute_path_name(): data_file_name: /dev/vx/dsk/gebdg/oracle_GEB_sapdata13
get_char_device_name Enter get_char_device_name(): ident_p->abs_pathname: /dev/vx/dsk/gebdg/oracle_GEB_sapdata13
get_pdev_clar_data Enter get_pdev_clar_data(): pdevname: /dev/vx/rdsk/gebdg/oracle_GEB_sapdata13
get_pdev_symm_data Enter get_pdev_symm_data(): pdevname: /dev/vx/rdsk/gebdg/oracle_GEB_sapdata13
get_logical_volume_info abs_pathname: /dev/vx/rdsk/gebdg/oracle_GEB_sapdata13
get_map_vg_type Enter get_map_vg_type(): abs_pathname: /dev/vx/rdsk/gebdg/oracle_GEB_sapdata13
get_lv_header_info Enter get_lv_header_info(): gen_vg_name: gebdg gen_lv_name: oracle_GEB_sapdata13
emcLvmGetLvmType Enter emcLvmGetLvmType()
get_veritas_lv_header_info Enter get_veritas_lv_header_info(): lvname: oracle_GEB_sapdata13
emcMapPopen emcMapPopen: cmd: PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin; export PATH; vxprint -g gebdg -v oracle_GEB_sapdata13 2>&1
emcMapPopen emcMapPopen: FILE * = 10D820
emcMapPclose emcMapPclose FILE * = 10D820; RC = 0
GetHostDefaultFsType Enter GetHostDefaultFsType()
SymFileSystemShow Enter SymFileSystemShow(): fsname: /oracle/GEB/sapdata13 host_fs_type: SunOS UFS File
check_map_license Enter check_map_license()
get_map_vg_type Enter get_map_vg_type(): abs_pathname: /dev/vx/rdsk/gebdg/oracle_GEB_sapdata13
SymMapLvShow Enter SymMapLvShow(): lvname: /dev/vx/rdsk/gebdg/oracle_GEB_sapdata13
emcCheckMapLicense Enter emcCheckMapLicense()
SymMapLvShow Enter SymMapLvShow(): gen_vg_name: gebdg gen_lv_name: oracle_GEB_sapdata13
emcLvmGetLvmType Enter emcLvmGetLvmType()
VeritasLvShow Enter VeritasLvShow(): lvname: oracle_GEB_sapdata13
VeritasGetLvData Enter VeritasGetLvData(): lvname: oracle_GEB_sapdata13
VeritasLvScan Enter VeritasLvScan(): lvname: oracle_GEB_sapdata13
emcMapPopen emcMapPopen: cmd: PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin; export PATH; vxprint -g gebdg -hmvps oracle_GEB_sapdata13 2>&1
emcMapPopen emcMapPopen: FILE * = 10D820
emcMapPclose emcMapPclose FILE * = 10D820; RC = 0
VeritasAddLog Enter VeritasAddLog(): lv_ptr->lvname: oracle_GEB_sapdata13
VeritasAddMir Enter VeritasAddMir(): lv_ptr->lvname: oracle_GEB_sapdata13
VeritasAddExt Enter VeritasAddExt(): lv_ptr->lvname: oracle_GEB_sapdata13
SUNget_rdevice_data Enter SUNget_rdevice_data(): disk: /dev/rdsk/EMC0_102s2 slice: EMC0_102
SUNget_rdevice_data open failed with errno 2, /dev/rdsk/EMC0_102s2
VeritasAddExt get_rdevice_data failed, EMC0_102
VeritasAddMir NULL failed, Extent add failed
SymMapLvShow gebdg failed, LvShow failed
SymDataObjectResolve Function Completed; status = 168 (The attempt to interrogate a physical device associated with a volume group failed) (handle: 0)
The attempt to interrogate a physical device associated with a volume group failed
SymExit Function enter, handle: 0
SymExit Regress: DEBUG CAT is 0x00000001, INTERNAL_TRACE
SymExit Regress: DEBUG CAT is 0x00000002, FUNCTION_TRACE
SymExit Regress: DEBUG CAT is 0x00000004, SESSION_LOCKS
SymExit Regress: DEBUG CAT is 0x00000008, HOST_LOCKS
SymExit Regress: DEBUG CAT is 0x00000010, SYMINIT
SymExit Regress: DEBUG CAT is 0x00000020, SYMDISCOVER
SymExit Regress: DEBUG CAT is 0x00000040, LOAD
SymExit Regress: DEBUG CAT is 0x00000080, NET_CONNECTIONS
SymExit Regress: DEBUG CAT is 0x00000100, NET_FUNCTIONS
SymExit Regress: DEBUG CAT is 0x00000200, NET_ENCODE
SymExit Regress: DEBUG CAT is 0x00000400, NET_OTHER
SymExit Regress: DEBUG CAT is 0x00000800, OTHER
SymExit Regress: DEBUG CAT is 0x00001000, MAP_FS
SymExit Regress: DEBUG CAT is 0x00002000, MAP_DB
SymExit Regress: DEBUG CAT is 0x00004000, MAP_LVM
SymExit Regress: DEBUG CAT is 0x00008000, MAP_RSLV
SymExit Regress: DEBUG CAT is 0x00010000, SIL_CDB_BUFFERS
SymExit Regress: DEBUG CAT is 0x00020000, SIL_INTERNAL_TRACE
SymExit Regress: DEBUG CAT is 0x00040000, SIL_OTHER_MESSAGES
SymExit Regress: DEBUG CAT is 0x00080000, SIL_SYSCALLS
SymExit Regress: DEBUG CAT is 0x00100000, BASIC_CTRL
SymExit Regress: DEBUG CAT is 0x00200000, RDF
SymExit Regress: DEBUG CAT is 0x00400000, BCV
SymExit Regress: DEBUG CAT is 0x00800000, INLINES
SymExit Regress: DEBUG CAT is 0x01000000, OPTMZR
SymExit Regress: DEBUG CAT is 0x02000000, BCS
SymExit Regress: DEBUG CAT is 0x04000000, QOS
SymExit Regress: DEBUG CAT is 0x08000000, CFGCHG
SymExit Regress: DEBUG CAT is 0x10000000, SYMLOCK
SymExit Regress: DEBUG CAT is 0x20000000, SNAP
SymExit Regress: DEBUG CAT is 0x40000000, D0RUN
SymExit Regress: DEBUG2 CAT is 0x00000001, CHECKSUM
SymExit Regress: DEBUG2 CAT is 0x00000002, ACCESS_CONTROL
SymExit Regress: DEBUG2 CAT is 0x00000004, ACCESS_AUDIT
SymExit Regress: DEBUG2 CAT is 0x00000008, NETCONNECT_AUDIT
SymExit Regress: DEBUG2 CAT is 0x00000010, DEVMASK
SymExit Regress: DEBUG2 CAT is 0x00000020, DDF
SymExit Regress: DEBUG2 CAT is 0x00000040, CONNECTIVITY
SymExit Regress: DEBUG2 CAT is 0x00000080, EVENT
SymExit Regress: DEBUG2 CAT is 0x00000100, ERASE
SymExit Regress: DEBUG2 CAT is 0x00100000, RCOPY
SymExit Regress: DEBUG2 CAT is 0x3FF00000, CTRL_ALL
emcSymLookaside Lookaside release size: 656384
SymExit Function exit - session closed
This error seems to be similar to "emc148392" solution ID. But can you can see we are using the native names for the device. Hence the workaround would not work for us. Could you please suggest a way to resolve this issue?
JamesBEMC
257 Posts
0
February 1st, 2008 00:00
What version of Solutions Enabler are you running?
According to the debug either something in the structure of the VG is inconsistent in you VG DB or else it could be a resolution bug in SE.
James.
rahulkhare
4 Posts
0
February 1st, 2008 06:00
We were using 6.0.3 version of SEK. Hence thinking on the same terms we upgraded the SEK version to 6.3.2.20 and later on to 6.4.3.0. But this symrslv issue still persists and the errors are identically same with all three SEK versions. Seems like we need to re-check the Veritas Disk group config. Although, it appears to be fine but would not like to rule out any option.
BTW, could you tell me if there is any other software available which could do the
1. Generate the replica using online-split R1-BCV.
2. Emulate the filesystems on BCV devices similar to R1 devices.
Thanks for your help,
Rahul.