ECS: NFS Remote I/O errors; Change of bucket owner for FS enabled bucket may lead to applications and or users unable to access NFS files

Change made by user to the bucket owner page on the user interface:


This issue applies to NFS enabled buckets and a change of the bucket owner by the user interface. This may cause applications or users connected to loss access to the bucket on the Linux file system. Even if we revert the change back to the original owner again access will not be possible leading to DU.

In this example:
The bucket owner was changed to "sham2" using the user interface. Due to a limitation in ECS, even after changing the bucket owner name back to "sham1." The ECS will not change the bucket owner back to "sham1" using the user interface. This can only be done for now with the CLI using an API with payload to resetowner flag to true.

Ways to identify the issue, on the Linux machine ask user to touch a file for example:

admin@node1~>touch file
touch: setting times of `file': Remote I/O error

se svc_log with the string  "method updateObjectInternal " 

Command:
# svc_log -a -sr dataheadsvc | grep "method updateObjectInternal"
Example:

admin@node1~>svc_log -a -sr dataheadsvc | grep "method updateObjectInternal"
svc_log v1.0.22 (svc_tools v1.5.3)                 Started 2019-06-06 10:45:04

Running on nodes:              <All nodes>
Time range:                    2019-06-05 10:45:04 - 2019-06-06 10:45:04
Filter string(s):              <All messages>
Show nodename(s):              True
Search reclaim logs (if any):  False

com.emc.storageos.data.object.exception.ObjectControllerException: method updateObjectInternal not allowed for previous bucket owner sham1
Caused by: com.emc.storageos.data.object.exception.ObjectControllerException: method updateObjectInternal not allowed for previous bucket owner sham1

Creating a bucket with a specific user as owner, followed by a change of the ownership of the bucket. Lastly, giving the original owner full control using the ACLs page fails with the ECS log exception:

ObjectControllerException: method updateObjectInternal not allowed for previous bucket owner <ownerid>
This is a known issue currently being evaluated by Dell EMC at this time.

The workaround is to change the bucket owner using the API through CLI with payload to resetowner flag to true.

1. Determine the current bucket owner.

admin@ecsnode1:~> tok=$(curl -iks https://XX.XX.XX.XX:4443/login -u 'root:ChangeMe' | grep X-SDS-AUTH-TOKEN)

admin@node1:~> curl -s -k -X GET -H "$tok" https://XX.XX.XX.XX:4443/object/bucket/sham_bk_nfs/info?namespace=degreat_nfs | xmllint --format - | grep '<owner>'
  <owner>sham2</owner>

2. Create a simple xml file by using the vi editor. In the example below it is called,  /tmp/bucket-owner.xml. This, is a two-step process. We have to set it to a new owner temporarily of sham2. As in this example below, before setting back to original owner sham1, confirm the output:

admin@node1:~ # vi /tmp/bucket-owner.xml

admin@ecsnode1:~ # cat /tmp/bucket-owner.xml
<object_bucket_update_owner>
   <namespace>degreat_nfs</namespace>
   <new_owner>sham2</new_owner>
   <reset_previous_owners>true</reset_previous_owners>
</object_bucket_update_owner>

3. Change the bucket owner to the temporary owner. 

admin@ecsnode1:~> curl -v -k -X "POST" "https://xx.xx.xx.xx:4443/object/bucket/sham_bk_nfs/owner" -H "$tok" -H "Content-Type: application/xml" -H "ACCEPT:application/xml" -d @/tmp/bucket-owner.xml -v
* Hostname was NOT found in DNS cache
*   Trying xx.xx.xx.xx...
* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 4443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* SSLv3, TLS unknown, Certificate Status (22):
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, Certificate (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*        subject: CN=localhost
*        start date: 2019-03-25 09:53:41 GMT
*        expire date: 2029-03-22 09:53:41 GMT
*        issuer: CN=localhost
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> POST /object/bucket/sham_bk_nfs/owner HTTP/1.1
> User-Agent: curl/7.37.0
> Host: xx.xx.xx.xx:4443
> X-SDS-AUTH-TOKEN: BAAcUy9KYlhxTlVYb2M0bnF3bTNscEsvSEdDeWhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmJhOGQ3ZTkzLTMyMGYtNDNmNy05Y2FkLWM4YWQzMWFiMzY1MAIADTE1NTk3Mzk3OTA2MDgDAC51cm46VG9rZW46YjQ4NGNiZjEtNTkwNy00YWI3LTgzYTctM2Y3OGRhM2RiY2NiAgAC0A8=
> Content-Type: application/xml
> ACCEPT:application/xml
> Content-Length: 179
>
* upload completely sent off: 179 out of 179 bytes
< HTTP/1.1 200 OK
< Date: Thu, 06 Jun 2019 10:56:08 GMT
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host xx.xx.xx.xx left intact
4. Edit the simple.xml file previously created in step 2 and this time insert original owner of sham1

admin@node1:~ # vi /tmp/bucket-owner.xml

admin@ecsnode1:~ # cat /tmp/bucket-owner.xml
<object_bucket_update_owner>
   <namespace>degreat_nfs</namespace>
   <new_owner>sham1</new_owner>
   <reset_previous_owners>true</reset_previous_owners>
</object_bucket_update_owner>

5. Change the bucket owner back to the original owner
The API syntax required to change the bucket owner back to "sham1" through the xml file is as follows:

admin@ecsnode1:~> curl -v -k -X "POST" "https://xx.xx.xx.xx:4443/object/bucket/sham_bk_nfs/owner" -H "$tok" -H "Content-Type: application/xml" -H "ACCEPT:application/xml" -d @/tmp/bucket-owner.xml -v
* Hostname was NOT found in DNS cache
*   Trying xx.xx.xx.xx...
* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 4443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* SSLv3, TLS unknown, Certificate Status (22):
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, Certificate (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*        subject: CN=localhost
*        start date: 2019-03-25 09:53:41 GMT
*        expire date: 2029-03-22 09:53:41 GMT
*        issuer: CN=localhost
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> POST /object/bucket/sham_bk_nfs/owner HTTP/1.1
> User-Agent: curl/7.37.0
> Host: xx.xx.xx.xx:4443
> X-SDS-AUTH-TOKEN: BAAcUy9KYlhxTlVYb2M0bnF3bTNscEsvSEdDeWhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmJhOGQ3ZTkzLTMyMGYtNDNmNy05Y2FkLWM4YWQzMWFiMzY1MAIADTE1NTk3Mzk3OTA2MDgDAC51cm46VG9rZW46YjQ4NGNiZjEtNTkwNy00YWI3LTgzYTctM2Y3OGRhM2RiY2NiAgAC0A8=
> Content-Type: application/xml
> ACCEPT:application/xml
> Content-Length: 179
>
* upload completely sent off: 179 out of 179 bytes
< HTTP/1.1 200 OK
< Date: Thu, 06 Jun 2019 10:56:08 GMT
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host xx.xx.xx.xx left intact


6.  Confirm the bucket owner change is reflected. 

admin@ecsnode1:~> curl -s -k -X GET -H "$tok" https://XX.XX.XX.XX:4443/object/bucket/sham_bk_nfs/info?namespace=degreat_nfs | xmllint --format - | grep '<owner>'
<owner>sham1</owner>

. 7. Once the config change is done, we should not see the same error any more

svc_log -f "method updateObjectInternal not allowed" -start "20 hour ago" -sr all -sh -st hour
svc_log v1.0.22 (svc_tools v1.6.8)                 Started 2020-01-23 09:28:17

Running on nodes:              <All nodes>
Time range:                    2020-01-22 13:28:17 - 2020-01-23 09:28:17
Filter string(s):              'method updateObjectInternal not allowed'
Show nodename(s):              True
Search reclaim logs (if any):  False

Count of message occurrences per hour:

2020-01-22 13:xx - 5066
2020-01-22 14:xx - 9580
2020-01-22 15:xx - 9574
2020-01-22 16:xx - 9580
2020-01-22 17:xx - 9570
2020-01-22 18:xx - 9576
2020-01-22 19:xx - 9564
2020-01-22 20:xx - 9576
2020-01-22 21:xx - 9576
2020-01-22 22:xx - 9572
2020-01-22 23:xx - 9564
2020-01-23 00:xx - 9586
2020-01-23 01:xx - 9574
2020-01-23 02:xx - 9572
2020-01-23 03:xx - 4564
2020-01-23 04:xx - 0
2020-01-23 05:xx - 0
2020-01-23 06:xx - 0
2020-01-23 07:xx - 0
2020-01-23 08:xx - 0
2020-01-23 09:xx - 0


Dell EMC is aware of this issue and are working on a fix in a future release.

Related NFS KB:

• ECS: How to create a basic NFS export and mount it on a client
• ECS: dataheadsvc log streaming error: NFSv3 procedure LINK not supported in request ReadLinkRequest
• ECS: NFS mount failing with No such file or directory or ERROR_OBJECT_NOT_FOUND
• ECS: Sub folder or directory created using S3 Browser is not listed by NFS Client/Linux machine but sub folder content appears in main or root directory
• ECS: NFS Remote I/O errors; Change of bucket owner for FS enabled bucket may lead to applications/users unable to access NFS files
• ECS: NFS write throws I/O error after a certain amount of data.
• ECS: Using NFS file share from ECS with a VMware NFS datastore
• ECS: Best practices for mounting ECS NFS exports
• ECS: How to mount NFS share on Windows client
• ECS: NFS fails to mount after changing file export settings in the UI
• ECS: Is Oracle WebCenter Content (WCC) compatible with ECS?

Subscribe to product updates.
You can subscribe to updates by following the instructions in the Knowledge Article below:
DELL EMC: How to subscribe to Product Pages - Dell Support?