Highlighted
nogs1
1 Copper

.NET API and Checksum Hashing Algorithms

I've been playing around with the .NET API and am looking at using the MD5 checksum algorithm rather than SHA0 as we have integrations with other systems that use MD5 already.  Atmos version is 2.1 and a simple/small upload with an MD5 checksum appears to work, but when the data is larger and the checksum is calculated by using multiple 64KB buffers the checksum method throws an exception.

What I think is happening is that the MD5, and probably the SHA1 version as well, are using the .NET implementations which appear to be non-managed classes.  When the hash object gets deep copied into the finalizedHash object something isn't getting copied properly and this is probably causing the issue.  

Say you have a 100KB file to upload and you are uploading in one go, i.e. not chunking but this would still be affected.   The checksum will get calculated in 2 parts, the first 64KB then a second buffer of the remaining data.   The first 64KB appears to get calculated fine, but trying to add the second buffer to the checksum fails with a System.Security.Cryptography.CryptographicException "Hash not valid for use in specified state."  Full trace is:

   at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)

   at System.Security.Cryptography.Utils.HashData(SafeHashHandle hHash, Byte[] data, Int32 cbData, Int32 ibStart, Int32 cbSize)

   at System.Security.Cryptography.MD5CryptoServiceProvider.HashCore(Byte[] rgb, Int32 ibStart, Int32 cbSize)

   at System.Security.Cryptography.HashAlgorithm.TransformBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[] outputBuffer, Int32 outputOffset)

   at EsuApiLib.Checksum.Update(ArraySegment`1 data) in c:\Work\atmos-dotnet\EsuApiLib\Checksum.cs:line 80

   at EsuApiLib.Rest.EsuRestApi.CreateObjectFromStreamOnPath(ObjectPath path, Acl acl, MetadataList metadata, Stream data, Int64 streamLength, String mimeType, Checksum checksum) in c:\Work\atmos-dotnet\EsuApiLib\Rest\EsuRestApi.cs:line 612

   at Clacks.Main.uploadSmall_Click(Object sender, EventArgs e) in c:\Work\Clacks\src\Clacks\Main.cs:line 131

   at System.Windows.Forms.Control.OnClick(EventArgs e)

   at System.Windows.Forms.Button.OnClick(EventArgs e)

   at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)

   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)

   at System.Windows.Forms.Control.WndProc(Message& m)

   at System.Windows.Forms.ButtonBase.WndProc(Message& m)

   at System.Windows.Forms.Button.WndProc(Message& m)

   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)

   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)

   at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

   at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)

   at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)

   at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)

   at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)

   at System.Windows.Forms.Application.Run(Form mainForm)

   at Clacks.Program.Main() in c:\Work\Clacks\src\Clacks\Program.cs:line 25

   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)

   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)

   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()

   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Threading.ThreadHelper.ThreadStart()

I've tried a couple of different deep copy algorithms on the .net MD5 hash algorithm objects, MD5Cng and MD5CryptoServiceProvider, and it seems to copy fine, but will fail the next time TransformBlock is called on the original object after TransformFinalBlock is called on the clone.  I also tried with the .net SHA256Managed class and interesting that worked fine so that is why I think it is probably the non-managed aspect of the MD5 and SHA1 providers that is causing the deep copy problem.

I tested the MD5 class from the library that is used for the SHA0 and that appears to clone fine.  Would a short-term fix be to change the MD5 and SHA1 algortihms to use the external library versions?

Also, I've made a local change to the UploadHelper class to make the checksum object public so that as a client we can see what the calculated checksum is without having to check it again or make a request from the server something like as I couldn't see another way to get the checksum as the .net API doesn't appear to add the property to the metadata if you call GetSystemMetadata:

ReadObjectStreamResponse response = esuApi.ReadObjectStream( op, new Extent( 0, 1 ) );

Console.WriteLine( response.ContentChecksum );

Thanks,

Steve

Labels (2)
0 Kudos
1 Reply
8 Krypton

Re: .NET API and Checksum Hashing Algorithms

Hi Steve,

Our apologies, your post appears to have gotten lost somewhere in our moderation queue and just appeared now.  Are you still having this issue?  We have not seen this on our side; can you provide some more details about your environment or provide a sample application?

0 Kudos