UNSOLVED
Cloudpools Question
I am testing cloudpools. I was able to add the cloud pools from Azure and AWS and my filepool polices are working fine. I see the files archived to the cloud as stubbed. Here are two problems I have:
1. I don't see the files from the Azure or S3 side. I am an admin on AWS and I still can't not see the files. I made a a read policy for myself on the S3 and still no luck.
2. When I try to open the files that were moved from the SMB shares, It fails..saying "can not open....make sure a disk is in the drive?. Do we have to recall the files first from OneFS to open the stubbed files?
Just want to know if this is by design or am I missing anything? I am testing on a virtual cluster running OneFS 8.0.0.4.
Thanks
-Daniel
Phil,
I login to aws and check my S3 bucket , as well as tried using cloudberry. I don’t see the files I archived using cloud-pools but I see the rest of the files I moved using cloudberry or directly upload to the bucket directly by browsing to the S3 bucket.
I understand they are stubbed. I expect opening the files to be slow but am getting error opening them. I am assuming the same original SMB ACLs apply on the files even after they are archived. Correct?
Thanks
Correct on SMB ACLs.
Are there any m000.... or d000.... folder in the AWS bucket? Can you give an output of "isi cloud accounts list"?
example with ECS:
No, we don't have any folders under the bucket.
isilon-1# isi cloud accounts list
Name Type Account Username URI State
----------------------------------------------------------------------------------------------------------------------
aws-s3-archive s3 my_aws_account https://s3.console.aws.amazon.com/s3/buckets/my_aws_bucket OK
azure-blob-archive azure my_azure_account https://my_azure_account.blob.core.windows.net/ OK
----------------------------------------------------------------------------------------------------------------------
Total: 2
isilon-1# isi cloud pools list
Name Type State
----------------------------
aws-s3-pool s3 OK
azure-blob-pool azure OK
----------------------------
Total: 2
isilon-1#
Somehow the m000.... & d000... folders got deleted. Isilon cloudpools stores the encrypted data in 1MB chunks in there. m000 .. for metadata and d000 for data/files chunks.
Try creating another bucket and see if those folders are there.
Phil,
Are the m000.. and d000..folders under the bucket created when we connect to the bucket for the first time or when we start archiving to it?
Also if we start archiving from different paths on the Isilon to the same bucket , will all be under the same d000...? Meaning will every target bucket have its own d000..and m000..folders?
Thank you for your help,
-Dan
Are the m000.. and d000..folders under the bucket created when we connect to the bucket for the first time or when we start archiving to it?
PHIL> yes
Also if we start archiving from different paths on the Isilon to the same bucket , will all be under the same d000...?
PHIL> yes, there will be only 1 m000.. and d000.. for all data archived
Phil,
I finally got it to work. I followed the document. I was using the S3 bucket URI not the main AWS URI. Thank you for your help.
One question...I archived a directory size of 3.2 GB. After archival, the size went to 128K or so. I went ahead and recalled a 1GB file but the size of the directory did not increase by 1GB. It still remained 128K although the recall job completed. Any idea why?
Thank you for all your help,
-Daniel
I will be patient..:)
I see this status about the job so I assumed it did finish:
isilon-1# isi cloud jobs list| grep -i recall
244 completed recall
isilon-1# isi cloud jobs view 244
ID: 244
Description:
Effective State: completed
Type: recall
Operation State: running
Job State: completed
Create Time: 2018-06-22T16:47:25
State Change Time: 2018-06-22T16:55:07
Completion Time: 2018-06-22T16:55:07
Job Engine Job: 2169
Job Engine State: succeeded
Total Files: 1
Total Canceled: 0
Total Failed: 0
Total Pending: 0
Total Staged: 0
Total Processing: 0
Total Succeeded: 1
isilon-1#
Phil.Lam
4 Apprentice
•
641 Posts
•
12 Points
2924
0
Posted June 19th, 2018 15:00
Dtek
1. are you using S3 browser or Cyberduck to check files on cloud provider.
2. the files are stubbed and has to reach the cloud provider to read the contents.