Atmos 1.4 wrappers now available

Hi Jason, can you expand a little bit more on the features related to CheckSums?

Do the new APIs allow you to return a checksum value on write? Under the PHP_Api updates, it looks like you can checksum the erasure coded replicas? How does that work?

Thanks for all the work on the APIs. Looking forward to trying them out

Saunders

Hi Saunders, sorry it took me a while to get back to this.

Checksums are supported during write operations for all objects.  You simply create a new Checksum object and pass it along with the create request.  If you're creating an object and then appending chunks to it, you should reuse the same checksum object for subsequent requests, e.g.

     Checksum ck = new Checksum( Algorithm.SHA0 );
     ObjectId id = this.esu.createObject( null, null, "hello".getBytes( "UTF-8" ), "text/plain", ck );

     this.esu.updateObject( null, null, new Extent( 5, 6 ), " world".getBytes( "UTF-8" ), null, ck );

When reading an object back, checksums are only supported for objects with erasure coded replicas.  As before, create a new checksum object and pass it along with all your read requests.  When you've read the complete object, compare the expected value with the actual value:

    if(  checksum.getExpectedValue() != null ) {
        // Validate
        if( !checksum.getExpectedValue().equals( checksum.toString() ) ) {
            throw new EsuException("Checksum validation error.  Expected " + checksum.getExpectedValue() + " but computed " + checksum.toString() );
        } else {
            l4j.info( "Checksum OK: " + checksum.getExpectedValue() );
        }

    }

If the object is not erasure coded, the expected value of the checksum will be null.

Hello, Jason.  Any guidance what size 'chunks' when creating/ updating a large file?

Hi Chris,

It depends on your application and connection speed, but I'd recommend between 1MB and 4MB chunks.

When reading files, you'll get the most memory efficiency by using 1MB chunks.  Above 1MB, Atmos will issue a Content-Encoding:Chunked response without Content-Length so we have to buffer the response since we're not absolutely sure how big it is.

Hi Jason,

I having problems running the code on incremental checksum validation for update chunks.

For example this:

                Checksum ck = new Checksum(Checksum.Algorithm.SHA0);

                ObjectId id = esu.CreateObject(null, null, Encoding.UTF8.GetBytes("hello"), "text/plain", ck);

                esu.UpdateObject(id, null, null, new Extent(5, 6), Encoding.UTF8.GetBytes(" world"), null, ck);

                Console.WriteLine(Encoding.UTF8.GetString(esu.ReadObject(id, null, null)));

throws this exception:

System.ArgumentException: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.

   at System.Buffer.BlockCopy(Array src, Int32 srcOffset, Array dst, Int32 dstOffset, Int32 count)

   at Home.Andir.Cryptography.BlockHashAlgorithm.HashCore(Byte[] array, Int32 offset, Int32 length)

   at System.Security.Cryptography.HashAlgorithm.TransformBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[] outputBuffer, Int32 outputOffset)

   at EsuApiLib.Checksum.Update(ArraySegment`1 data)

   at EsuApiLib.Rest.EsuRestApi.UpdateObjectFromSegment(Identifier id, Acl acl,MetadataList metadata, Extent extent, ArraySegment`1 data, String mimeType, Checksum checksum)

   at EsuApiLib.Rest.EsuRestApi.UpdateObject(Identifier id, Acl acl, MetadataList metadata, Extent extent, Byte[] data, String mimeType, Checksum checksum)

   at Console_UpdateObject.Program.Main(String[] args)

Looking the file of Cheksum.cs class, the bold line is the one that throws the exception:

        public void Update( ArraySegment data ) {

            finalizedHash = (HashAlgorithm)deepCopy(hash);

            hash.TransformBlock(data.Array, data.Offset, data.Count, null, 0);

            // We do this here (instead of a null transform) because the SHA0

            // implementation seems to fail for short messages if you simply

            // transform an empty block later.

            finalizedHash.TransformFinalBlock(data.Array, data.Offset, data.Count);

            offset += data.Count;

        }

Te version of the EsuApiLib.dll is 1.4.1.16.

Is there some kind of consideration on incremental object updates to support checksum validation?

Yes, the checksum feature is designed to support incremental append operations.  Your code looks correct and I see the same error here.  I suspect there may be an error in the underlying hash algorithm when handling multiple blocks that are smaller than the hash's block size.  I will investigate and get back to you.

Created an issue and fixed it.  Version 1.4.1.23 is released.

http://code.google.com/p/atmos-dotnet/downloads/detail?name=EsuDotNet_1.4.1.23.zip&can=2&q=

Edited 12/20/11 - corrected version and file name.

Thanks!!!