Cloudpools Question

Dtek

1. are you using S3 browser or Cyberduck to check files on cloud provider.

2. the files are stubbed and has to reach the cloud provider to read the contents.

Daniel,

Correct on SMB ACLs.

Are there any m000.... or d000.... folder in the AWS bucket? Can you give an output of  "isi cloud accounts list"?

example with ECS:

Isilon OneFS 8.0 CloudPools and ECS demo

No, we don't have any folders under the bucket.

isilon-1# isi cloud accounts list

Name               Type  Account Username     URI                                                               State

----------------------------------------------------------------------------------------------------------------------

aws-s3-archive     s3    my_aws_account              https://s3.console.aws.amazon.com/s3/buckets/my_aws_bucket OK

azure-blob-archive azure my_azure_account        https://my_azure_account.blob.core.windows.net/               OK

----------------------------------------------------------------------------------------------------------------------

Total: 2

isilon-1# isi cloud pools list

Name            Type  State

----------------------------

aws-s3-pool     s3    OK

azure-blob-pool azure OK

----------------------------

Total: 2

isilon-1#

Daniel,

Somehow the m000.... & d000... folders got deleted.  Isilon cloudpools stores the encrypted data in 1MB chunks in there.  m000 .. for metadata and d000 for data/files chunks.

Try creating another bucket and see if those folders are there.

Phil,

Are the m000.. and d000..folders under the bucket created when we connect to the bucket for the first time or when we start archiving to it?

Also if we start archiving from different paths on the Isilon to the same bucket , will all be under the same d000...? Meaning will every target bucket have its own d000..and m000..folders?

Thank you for your help,

-Dan

Daniel,

Are the m000.. and d000..folders under the bucket created when we connect to the bucket for the first time or when we start archiving to it?

PHIL> yes

Also if we start archiving from different paths on the Isilon to the same bucket , will all be under the same d000...?

PHIL> yes, there will be only 1 m000.. and d000.. for all data archived

Phil,

I finally got it to work. I followed the document. I was using the S3 bucket URI not the main AWS URI. Thank you for your help.

One question...I archived a directory size of 3.2 GB. After archival, the size went to 128K or so. I went ahead and recalled a 1GB file but the size of the directory did not increase by 1GB. It still remained 128K although the recall job completed. Any idea why?

Thank you for all your help,

-Daniel

I will be patient..:)

I see this status about the job so I assumed it did finish:

isilon-1# isi cloud jobs list| grep -i recall

244                                          completed        recall

isilon-1# isi cloud jobs view 244

               ID: 244

      Description:

  Effective State: completed

             Type: recall

  Operation State: running

        Job State: completed

      Create Time: 2018-06-22T16:47:25

State Change Time: 2018-06-22T16:55:07

  Completion Time: 2018-06-22T16:55:07

   Job Engine Job: 2169

Job Engine State: succeeded

      Total Files: 1

   Total Canceled: 0

     Total Failed: 0

    Total Pending: 0

     Total Staged: 0

Total Processing: 0

  Total Succeeded: 1

isilon-1#

Dtek

Glad I could help. I'm guessing that it hasn't finished recalling the file yet.

Daniel,

Cloud Recall

check the stub flag using the command, isi get –D, to verify that the stub file has been recalled and become a regular file again.

# isi get -D /ifs/data/cp_test/cp_testfile.txt | grep -i stub

* Stubbed: False

The Stubbed Flag is marked “False”, indicating that it’s a regular file.

or on Isilon "ls -lo ". If it stubbed there is a sstubbed tag on the file. If it's regular file, no sstubbed tag.

Phil,

isilon-1# ls -ld amazon

drwxrwx--- +  2 root  wheel  215 Jun 22 16:28 amazon

isilon-1# isi get -D amazon/test_file | grep -i stub

*  Stubbed:            False

isilon-1# ls -loh amazon/test_file

-rwxrwx--- +  1 200048079  200048079  uarch,inherit,writecache,wcinherit,shasntfsacl,hasads  1.0G Jun 22 16:26 amazon/test_file

isilon-1# du -sh amazon

128K amazon

isilon-1#

As you can see the file is not stubbed. It is a 1GB file but the directory is still showing 128K. Before I archived the 'amazon' folder , it was 3.2GB.  128KB after the archive. But still remained same after I recalled the 1GB file.

Thanks

-Daniel

Daniel,

Check out KB 305012. Create an Advisory quota on that directory and it will give the size difference also.

NOTE: my quota was a hard quota

OneFS: How to use the du command to compare the space used to store and protect data https://support.emc.com/kb/305012

from my lab box:

Before files were stubbed in /ifs/data/CloudPool_ECS directory.

s200-1# du -sSHAk /ifs/data/CloudPool_ECS

133292  /ifs/data/CloudPool_ECS

s200-1# isi quota quotas list

Type      AppliesTo  Path                    Snap  Hard  Soft  Adv  Used

---------------------------------------------------------------------------

directory DEFAULT    /ifs/data/CloudPool_ECS No    2.00T -     -    130.17M

---------------------------------------------------------------------------

After files were stubbed in /ifs/data/CloudPool_ECS directory.

s200-1# du -sSHAk /ifs/data/CloudPool_ECS

4       /ifs/data/CloudPool_ECS

s200-1# isi quota quotas list

Type      AppliesTo  Path                    Snap  Hard  Soft  Adv  Used

-------------------------------------------------------------------------

directory DEFAULT    /ifs/data/CloudPool_ECS No    2.00T -     -    3.1k

-------------------------------------------------------------------------

Total: 1

Thank you Phil. The recall finally worked and I see the space going down as I bring files back from the cloud.

Have you ever tested cloudpooling data being SyncIQ'ed from source to target? Once the source data is archived to cloud, I ran the syncIQ job and it failed. I was expecting the target files to be replaced by smartlinks.  I see the cloudpool account copied over to taget but it is showing as disabled. "Account disabled until added to access list"

Should the cloud accounts be manually created on the target?

Dtek

Is the DR cluster licensed for SmartPools and CloudPools? You need it licensed for AWS. Does DR cluster have network access to AWS?

SyncIQ will copy over the accounts, but you have to have network connectivity to work correctly.

Phil