Data Domain Boost - Open Storage: Intermittent NetBackup errors 2060001 in log with DDBoost over FC
Summary: Random backup is failing with this error. Single stream is working.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
Backups are running very unstable.. Often we have an error code "83" in Netbackup which means "Media open error".
11/25/2016 02:37:26 - Critical bptm (pid=3074) sts_get_server_prop_byname failed: error 2060001 one or more invalid arguments
11/25/2016 02:37:26 - Critical bptm (pid=3074) failure to open sts for storage server DFC-XXXXXXXX: plug-in reports error 2060001 one or more invalid arguments
Cause
Check the Netbackup bptm log.
If you find "busy" as shown and the message is "No paths discovered to DFC server":
02:29:56.719 [3074] <4> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c8t5002188136612175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c8t5002188136612175d0 failed, error 16: Device busy
02:29:56.719 [3074] <4> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c8t5002188136612175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c8t5002188136612175d0 failed, error 16: Device busy
02:29:56.720 [3074] <4> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c11t5002188136602175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c11t5002188136602175d0 failed, error 16: Device busy
02:29:56.720 [3074] <4> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c11t5002188136602175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c11t5002188136602175d0 failed, error 16: Device busy
02:29:56.720 [3074] <8> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO: No paths discovered to DFC server XXXXXX
All devices are busy, the plugin didn't found a device available for new backup.
If you find "busy" as shown and the message is "No paths discovered to DFC server":
02:29:56.719 [3074] <4> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c8t5002188136612175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c8t5002188136612175d0 failed, error 16: Device busy
02:29:56.719 [3074] <4> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c8t5002188136612175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c8t5002188136612175d0 failed, error 16: Device busy
02:29:56.720 [3074] <4> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c11t5002188136602175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c11t5002188136602175d0 failed, error 16: Device busy
02:29:56.720 [3074] <4> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c11t5002188136602175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c11t5002188136602175d0 failed, error 16: Device busy
02:29:56.720 [3074] <8> 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO: No paths discovered to DFC server XXXXXX
All devices are busy, the plugin didn't found a device available for new backup.
Resolution
You need more devices or limit the number of backups in parallel.
This is documented in the DDBOOST Admin Guide: https://support.emc.com/docu61771_Data-Domain-Boost-for-OpenStorage-3.2.1-Administration-Guide.pdf?language=en_US
"Solaris Media Servers For Solaris, device entries are exclusively locked on a per-process basis one and only one process can use a device entry. Calculations are based on application instance usage. If an application spawns multiple processes, each process exclusively locks at least one device entry. Multi-threaded applications lock one device per thread. For these reasons, you should configure the Data Domain system to advertise as many DFC devices as possible to avoid 'in use' errors from the sgen device driver. A Device Busy error may result if there are not enough devices accessible to the Solaris clients. The number of sgen devices is the number of Fibre Channel ports accessible to the Solaris instance times the number of different paths to the Data Domain system endpoint(s) times the number of LUNs in the access group."
"Windows Media Servers : The Data Domain server path management logic spreads out connections across available logical paths (Initiator, Target Endpoint, DFC Device). Configure enough DFC devices such that each connection uses its own generic SCSI device (logical path) on the media server, with a max DFC device count of 64.
certain client-side operating systems impose a restriction on the number of outstanding IO requests which can be conducted simultaneously over a given generic SCSI device. For example, the Windows SCSI Pass-Through Interface mechanism will only conduct 1 SCSI request at a time through each of its generic SCSI devices. This impacts the performance of the DD Boostover FC solution, if multiple connections (e.g. backup jobs) are trying to use the same generic SCSI device. Additionally, the Data Domain system also imposes a limit on the number of outstanding IO requests per advertised SCSI device. For performance reasons with larger workloads,
multiple SCSI devices may need to be advertised on the Data Domain system. We use the term queue-depth to describe the system-imposed limit on the number of simultaneous SCSI requests on a single device. Client systems (like Windows) whose queue depth is so low as to impact performance are considered queue-depth constrained.
This is documented in the DDBOOST Admin Guide: https://support.emc.com/docu61771_Data-Domain-Boost-for-OpenStorage-3.2.1-Administration-Guide.pdf?language=en_US
"Solaris Media Servers For Solaris, device entries are exclusively locked on a per-process basis one and only one process can use a device entry. Calculations are based on application instance usage. If an application spawns multiple processes, each process exclusively locks at least one device entry. Multi-threaded applications lock one device per thread. For these reasons, you should configure the Data Domain system to advertise as many DFC devices as possible to avoid 'in use' errors from the sgen device driver. A Device Busy error may result if there are not enough devices accessible to the Solaris clients. The number of sgen devices is the number of Fibre Channel ports accessible to the Solaris instance times the number of different paths to the Data Domain system endpoint(s) times the number of LUNs in the access group."
"Windows Media Servers : The Data Domain server path management logic spreads out connections across available logical paths (Initiator, Target Endpoint, DFC Device). Configure enough DFC devices such that each connection uses its own generic SCSI device (logical path) on the media server, with a max DFC device count of 64.
certain client-side operating systems impose a restriction on the number of outstanding IO requests which can be conducted simultaneously over a given generic SCSI device. For example, the Windows SCSI Pass-Through Interface mechanism will only conduct 1 SCSI request at a time through each of its generic SCSI devices. This impacts the performance of the DD Boostover FC solution, if multiple connections (e.g. backup jobs) are trying to use the same generic SCSI device. Additionally, the Data Domain system also imposes a limit on the number of outstanding IO requests per advertised SCSI device. For performance reasons with larger workloads,
multiple SCSI devices may need to be advertised on the Data Domain system. We use the term queue-depth to describe the system-imposed limit on the number of simultaneous SCSI requests on a single device. Client systems (like Windows) whose queue depth is so low as to impact performance are considered queue-depth constrained.
Affected Products
Data Domain Boost - Open StorageProducts
Data Domain, Data Domain Boost - Open StorageArticle Properties
Article Number: 000059709
Article Type: Solution
Last Modified: 25 Sept 2024
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.