Unsolved
This post is more than 5 years old
79 Posts
0
5226
Cloudpools Question
I am testing cloudpools. I was able to add the cloud pools from Azure and AWS and my filepool polices are working fine. I see the files archived to the cloud as stubbed. Here are two problems I have:
1. I don't see the files from the Azure or S3 side. I am an admin on AWS and I still can't not see the files. I made a a read policy for myself on the S3 and still no luck.
2. When I try to open the files that were moved from the SMB shares, It fails..saying "can not open....make sure a disk is in the drive?. Do we have to recall the files first from OneFS to open the stubbed files?
Just want to know if this is by design or am I missing anything? I am testing on a virtual cluster running OneFS 8.0.0.4.
Thanks
-Daniel
Phil.Lam
3 Apprentice
3 Apprentice
•
592 Posts
0
June 19th, 2018 15:00
Dtek
1. are you using S3 browser or Cyberduck to check files on cloud provider.
2. the files are stubbed and has to reach the cloud provider to read the contents.
Dtek1
79 Posts
0
June 19th, 2018 15:00
Phil,
I login to aws and check my S3 bucket , as well as tried using cloudberry. I don’t see the files I archived using cloud-pools but I see the rest of the files I moved using cloudberry or directly upload to the bucket directly by browsing to the S3 bucket.
I understand they are stubbed. I expect opening the files to be slow but am getting error opening them. I am assuming the same original SMB ACLs apply on the files even after they are archived. Correct?
Thanks
Phil.Lam
3 Apprentice
3 Apprentice
•
592 Posts
0
June 19th, 2018 16:00
Daniel,
Correct on SMB ACLs.
Are there any m000.... or d000.... folder in the AWS bucket? Can you give an output of "isi cloud accounts list"?
example with ECS:
Isilon OneFS 8.0 CloudPools and ECS demo
Dtek1
79 Posts
0
June 20th, 2018 07:00
No, we don't have any folders under the bucket.
isilon-1# isi cloud accounts list
Name Type Account Username URI State
----------------------------------------------------------------------------------------------------------------------
aws-s3-archive s3 my_aws_account https://s3.console.aws.amazon.com/s3/buckets/my_aws_bucket OK
azure-blob-archive azure my_azure_account https://my_azure_account.blob.core.windows.net/ OK
----------------------------------------------------------------------------------------------------------------------
Total: 2
isilon-1# isi cloud pools list
Name Type State
----------------------------
aws-s3-pool s3 OK
azure-blob-pool azure OK
----------------------------
Total: 2
isilon-1#
Phil.Lam
3 Apprentice
3 Apprentice
•
592 Posts
0
June 21st, 2018 10:00
Daniel,
Somehow the m000.... & d000... folders got deleted. Isilon cloudpools stores the encrypted data in 1MB chunks in there. m000 .. for metadata and d000 for data/files chunks.
Try creating another bucket and see if those folders are there.
Dtek1
79 Posts
0
June 21st, 2018 14:00
Phil,
Are the m000.. and d000..folders under the bucket created when we connect to the bucket for the first time or when we start archiving to it?
Also if we start archiving from different paths on the Isilon to the same bucket , will all be under the same d000...? Meaning will every target bucket have its own d000..and m000..folders?
Thank you for your help,
-Dan
Phil.Lam
3 Apprentice
3 Apprentice
•
592 Posts
0
June 21st, 2018 15:00
Daniel,
Are the m000.. and d000..folders under the bucket created when we connect to the bucket for the first time or when we start archiving to it?
PHIL> yes
Also if we start archiving from different paths on the Isilon to the same bucket , will all be under the same d000...?
PHIL> yes, there will be only 1 m000.. and d000.. for all data archived
Dtek1
79 Posts
0
June 22nd, 2018 14:00
Phil,
I finally got it to work. I followed the document. I was using the S3 bucket URI not the main AWS URI. Thank you for your help.
One question...I archived a directory size of 3.2 GB. After archival, the size went to 128K or so. I went ahead and recalled a 1GB file but the size of the directory did not increase by 1GB. It still remained 128K although the recall job completed. Any idea why?
Thank you for all your help,
-Daniel
Dtek1
79 Posts
0
June 22nd, 2018 14:00
I will be patient..:)
I see this status about the job so I assumed it did finish:
isilon-1# isi cloud jobs list| grep -i recall
244 completed recall
isilon-1# isi cloud jobs view 244
ID: 244
Description:
Effective State: completed
Type: recall
Operation State: running
Job State: completed
Create Time: 2018-06-22T16:47:25
State Change Time: 2018-06-22T16:55:07
Completion Time: 2018-06-22T16:55:07
Job Engine Job: 2169
Job Engine State: succeeded
Total Files: 1
Total Canceled: 0
Total Failed: 0
Total Pending: 0
Total Staged: 0
Total Processing: 0
Total Succeeded: 1
isilon-1#
Phil.Lam
3 Apprentice
3 Apprentice
•
592 Posts
0
June 22nd, 2018 14:00
Dtek
Glad I could help. I'm guessing that it hasn't finished recalling the file yet.
Phil.Lam
3 Apprentice
3 Apprentice
•
592 Posts
0
June 22nd, 2018 14:00
Daniel,
Cloud Recall
check the stub flag using the command, isi get –D, to verify that the stub file has been recalled and become a regular file again.
# isi get -D /ifs/data/cp_test/cp_testfile.txt | grep -i stub
* Stubbed: False
The Stubbed Flag is marked “False”, indicating that it’s a regular file.
or on Isilon "ls -lo ". If it stubbed there is a sstubbed tag on the file. If it's regular file, no sstubbed tag.
Dtek1
79 Posts
0
June 22nd, 2018 19:00
Phil,
isilon-1# ls -ld amazon
drwxrwx--- + 2 root wheel 215 Jun 22 16:28 amazon
isilon-1# isi get -D amazon/test_file | grep -i stub
* Stubbed: False
isilon-1# ls -loh amazon/test_file
-rwxrwx--- + 1 200048079 200048079 uarch,inherit,writecache,wcinherit,shasntfsacl,hasads 1.0G Jun 22 16:26 amazon/test_file
isilon-1# du -sh amazon
128K amazon
isilon-1#
As you can see the file is not stubbed. It is a 1GB file but the directory is still showing 128K. Before I archived the 'amazon' folder , it was 3.2GB. 128KB after the archive. But still remained same after I recalled the 1GB file.
Thanks
-Daniel
Phil.Lam
3 Apprentice
3 Apprentice
•
592 Posts
0
June 23rd, 2018 22:00
Daniel,
Check out KB 305012. Create an Advisory quota on that directory and it will give the size difference also.
NOTE: my quota was a hard quota
OneFS: How to use the du command to compare the space used to store and protect data https://support.emc.com/kb/305012
from my lab box:
Before files were stubbed in /ifs/data/CloudPool_ECS directory.
s200-1# du -sSHAk /ifs/data/CloudPool_ECS
133292 /ifs/data/CloudPool_ECS
s200-1# isi quota quotas list
Type AppliesTo Path Snap Hard Soft Adv Used
---------------------------------------------------------------------------
directory DEFAULT /ifs/data/CloudPool_ECS No 2.00T - - 130.17M
---------------------------------------------------------------------------
After files were stubbed in /ifs/data/CloudPool_ECS directory.
s200-1# du -sSHAk /ifs/data/CloudPool_ECS
4 /ifs/data/CloudPool_ECS
s200-1# isi quota quotas list
Type AppliesTo Path Snap Hard Soft Adv Used
-------------------------------------------------------------------------
directory DEFAULT /ifs/data/CloudPool_ECS No 2.00T - - 3.1k
-------------------------------------------------------------------------
Total: 1
Dtek1
79 Posts
0
June 25th, 2018 14:00
Thank you Phil. The recall finally worked and I see the space going down as I bring files back from the cloud.
Have you ever tested cloudpooling data being SyncIQ'ed from source to target? Once the source data is archived to cloud, I ran the syncIQ job and it failed. I was expecting the target files to be replaced by smartlinks. I see the cloudpool account copied over to taget but it is showing as disabled. "Account disabled until added to access list"
Should the cloud accounts be manually created on the target?
Phil.Lam
3 Apprentice
3 Apprentice
•
592 Posts
0
June 25th, 2018 16:00
Dtek
Is the DR cluster licensed for SmartPools and CloudPools? You need it licensed for AWS. Does DR cluster have network access to AWS?
SyncIQ will copy over the accounts, but you have to have network connectivity to work correctly.
Phil