Atmos-php EMC Storage Object Checksum Value

Thanks Jason! As always!

By the way, are you able to give some pointer at which section/function/script of Atmos-php that I could provide a checksum value to Atmos for validation? Is there checksum mechanism to follow? Such as must be MD5 or SHA1 etc.

Understand at Atmos end the MD5 checksum values are stored in Berkeley DB which is not exposing to external application.

Greatly appreciate your further enlightenment.

Thank you.

Sure, you use the same Checksum object when uploading to Atmos.  Atmos <2.1 supports only SHA0, but with 2.1 we added SHA1 and MD5.

        public function testCreateChecksumSHA1() {
                if($this->atmosMajorMinor < 2.1) {
                        $this->markTestSkipped("Requires Atmos >= 2.1.0");
                        return;
                }
                $ck = new Checksum( 'SHA1' );
                $id = $this->esu->createObject( null, null, 'hello', 'text/plain', $ck );
                $this->cleanup[] = $id;
                PHPUnit_Framework_Assert::assertTrue( strlen(''.$ck) > 0, 'Checksum is empty' );
        }

http://code.google.com/p/atmos-php/source/browse/trunk/src/EsuRestApiTest.php#1241

Then, when you read the object back, you should get the x-emc-wschecksum to validate.  You can also get it by reading the system metadata for the object.

Thanks Jason,

I have tested it works, but the correct syntax to used is 'checksum' instead of 'x-emc-wschecksum' at least for atmos-php, for reference.

                $onlinecs = $cs->getMetadata( 'checksum' )->getValue();

Due to the following code in EsuRestApi.php line #651

                if (! empty($value))

                        { $m = new Metadata( 'checksum', $value, false );  $meta->addMetadata( $m ); }

Thank you.

Jason Cwik 

Hi Jason,

I am experiencing updateObject challenge as it doesn't recognize upload resume as a object as follow:

1st attempt of upload obtained SHA0 value SHA0/0/f96cea198ad1dd5617ac084a3d92c6107708c0ef and partially completed

2nd attempt of resume upload submit same SHA0 value and hits with following error

PHP Fatal error:  Call to a member function update() on a non-object in /home/atmos-php/src/EsuRestApi.php on line 430

The input of 2nd attempt of resume upload as follow:

Extent: offset: 4194304 size: 236978176

SHA0: SHA0/0/f96cea198ad1dd5617ac084a3d92c6107708c0ef

The code hits error at $checksum->update( $data ); with non-object error

Are you able extend your help to give some pointer here?

P.S.: When I perform an upload resume on a non-SHA0-enable file with updateObject() it works without error.

Thank you.

The problem is that the UploadHelper currently isn't smart enough to resume an upload.  To properly resume an upload, you must:

1) Query the object's metadata to see what the current checksum algorithm, offset, and value are.

2) Re-initialize the checksum object.

3) Replay the source data into the checksum object up to the server's reported offset.

4) Validate that the local value matches the remote value.

5) Resume uploading from the offset, using the new Checksum object.

You could probably do steps 1-4 yourself and set the checksum object directly on the UploadHelper without setting computeChecksums = true.

Hi Jason Cwik ,

We are suffering upload performance issue with turning on the checksum feature for SHA0 (due to Atmos version) as follow:

EsuHelpers.php (at Line 77)

private $computeChecksums = true;

Upload result for a 116MB file took 14 minutes.

private $computeChecksums = false;

Upload result for a 235MB file took less than a minute.

As you can see from above, there are a huge different in both with and without SHA0 checksum feature.

Are you able to extend your insight how we could achieve both upload performance and at the same time able to validate the checksum value at both local and online object without re-download it back for calculation?

Is other API language able to achieve that, please advice, in case is the limitation of Atmos-php in handling the checksum feaure?

Thank you!

The upload test result for reference:

Without Checksum

2013-06-07 10:55:44 0MB/230MB 0%

2013-06-07 10:55:44 4MB/230MB 2%

2013-06-07 10:55:45 8MB/230MB 3%

2013-06-07 10:55:45 12MB/230MB 5%

2013-06-07 10:55:45 16MB/230MB 7%

2013-06-07 10:55:46 20MB/230MB 9%

2013-06-07 10:55:47 24MB/230MB 10%

2013-06-07 10:55:48 28MB/230MB 12%

2013-06-07 10:55:48 32MB/230MB 14%

2013-06-07 10:55:48 36MB/230MB 16%

2013-06-07 10:55:50 40MB/230MB 17%

2013-06-07 10:55:50 44MB/230MB 19%

2013-06-07 10:55:51 48MB/230MB 21%

2013-06-07 10:55:51 52MB/230MB 23%

2013-06-07 10:55:52 56MB/230MB 24%

2013-06-07 10:55:53 60MB/230MB 26%

2013-06-07 10:55:53 64MB/230MB 28%

2013-06-07 10:55:54 68MB/230MB 30%

2013-06-07 10:55:55 72MB/230MB 31%

2013-06-07 10:55:55 76MB/230MB 33%

2013-06-07 10:55:55 80MB/230MB 35%

2013-06-07 10:55:56 84MB/230MB 37%

2013-06-07 10:55:57 88MB/230MB 38%

2013-06-07 10:55:57 92MB/230MB 40%

2013-06-07 10:55:58 96MB/230MB 42%

2013-06-07 10:55:58 100MB/230MB 43%

2013-06-07 10:55:59 104MB/230MB 45%

2013-06-07 10:56:00 108MB/230MB 47%

2013-06-07 10:56:00 112MB/230MB 49%

2013-06-07 10:56:01 116MB/230MB 50%

2013-06-07 10:56:01 120MB/230MB 52%

2013-06-07 10:56:03 124MB/230MB 54%

2013-06-07 10:56:03 128MB/230MB 56%

2013-06-07 10:56:05 132MB/230MB 57%

2013-06-07 10:56:05 136MB/230MB 59%

2013-06-07 10:56:06 140MB/230MB 61%

2013-06-07 10:56:06 144MB/230MB 63%

2013-06-07 10:56:07 148MB/230MB 64%

2013-06-07 10:56:07 152MB/230MB 66%

2013-06-07 10:56:08 156MB/230MB 68%

2013-06-07 10:56:08 160MB/230MB 70%

2013-06-07 10:56:09 164MB/230MB 71%

2013-06-07 10:56:09 168MB/230MB 73%

2013-06-07 10:56:11 172MB/230MB 75%

2013-06-07 10:56:12 176MB/230MB 77%

2013-06-07 10:56:12 180MB/230MB 78%

2013-06-07 10:56:12 184MB/230MB 80%

2013-06-07 10:56:14 188MB/230MB 82%

2013-06-07 10:56:14 192MB/230MB 83%

2013-06-07 10:56:14 196MB/230MB 85%

2013-06-07 10:56:17 200MB/230MB 87%

2013-06-07 10:56:17 204MB/230MB 89%

2013-06-07 10:56:18 208MB/230MB 90%

2013-06-07 10:56:18 212MB/230MB 92%

2013-06-07 10:56:19 216MB/230MB 94%

2013-06-07 10:56:19 220MB/230MB 96%

2013-06-07 10:56:20 224MB/230MB 97%

2013-06-07 10:56:20 228MB/230MB 99%

2013-06-07 10:56:21 230MB/230MB 100%

With Checksum

2013-06-07 11:10:49 0MB/230MB 0%

2013-06-07 11:10:49 4MB/230MB 2%

2013-06-07 11:11:20 8MB/230MB 3%

2013-06-07 11:11:50 12MB/230MB 5%

2013-06-07 11:12:20 16MB/230MB 7%

2013-06-07 11:12:49 20MB/230MB 9%

2013-06-07 11:13:21 24MB/230MB 10%

2013-06-07 11:13:51 28MB/230MB 12%

2013-06-07 11:14:21 32MB/230MB 14%

2013-06-07 11:14:51 36MB/230MB 16%

2013-06-07 11:15:21 40MB/230MB 17%

2013-06-07 11:15:51 44MB/230MB 19%

2013-06-07 11:16:21 48MB/230MB 21%

2013-06-07 11:16:52 52MB/230MB 23%

2013-06-07 11:17:22 56MB/230MB 24%

2013-06-07 11:17:51 60MB/230MB 26%

2013-06-07 11:18:21 64MB/230MB 28%

2013-06-07 11:18:51 68MB/230MB 30%

2013-06-07 11:19:21 72MB/230MB 31%

2013-06-07 11:19:52 76MB/230MB 33%

2013-06-07 11:20:24 80MB/230MB 35%

2013-06-07 11:20:53 84MB/230MB 37%

2013-06-07 11:21:24 88MB/230MB 38%

2013-06-07 11:21:55 92MB/230MB 40%

2013-06-07 11:22:25 96MB/230MB 42%

2013-06-07 11:22:55 100MB/230MB 43%

2013-06-07 11:23:25 104MB/230MB 45%

2013-06-07 11:23:55 108MB/230MB 47%

2013-06-07 11:24:25 112MB/230MB 49%

  2013-06-07 11:24:55 116MB/230MB 50%

Are you on Atmos 2.1?  If so, please try "SHA1" or "MD5".  The problem is that SHA0 is not widely available as a native algorithm so it needs to be implemented in software.  PHP has some limitations with its integer math that compounds this issue.  Using the other algorithms should help since native implementations can be used.  If you're not yet on 2.1, using a different language should provide better checksumming performance even with SHA0.

Thanks Jason Cwik  as always!

Managed to use other language for the checksumming operation and saw significant improvement over the native PHP code handling of the checksum operation.

Also, managed to replay the file and make resume operation fully functional.

Cheers!

Can we get an official word as to whether SHA-1 is supported in Atmos 2.1?  The API documentation 300-013-493.pdf for 2.1 does not list SHA-1.

Yes, Atmos 2.1 supports SHA0, SHA1 and MD5 checksums.