Unsolved
This post is more than 5 years old
3 Posts
0
1945
Intermittent Performance Issue Retrieving From Centera
Hi all,
We have been experiencing intermittent performance issues when retrieving from our Centera device and I'm hoping someone can help us. We're storing about 8.5 million documents, most of which contain a couple of 100 KB images. I'm using the latest EMC Centera SDK and .Net wrapper. Many of our retrieves from the Centera device come back in a second or less, but every now and then (there doesn't seem to be a pattern, sometime it's the first one, the tenth, or the hundredth retrieve), a retrieve will take 90 seconds or more, even for a clip with just two 100 KB images in it. It eventually comes back with the document, but oftentimes our front-end times out waiting.
We were originally using the XAM.NET wrapper to interface with the Centera device. Thinking that the problem might lie in the XAM.NET wrapper, we swapped it out for the COSI .NET wrapper for Centera on our retrieves, but the problem remained. Our Centera access logic is located in a web service that our front-end apps call, but we've also tried pulling it out into a Windows app and hitting the Centera directly, but to no avail. We've also tried using a single pool for all requests as well as creating and then closing and disposing of the pool for each individual request - both approaches yielded the same result - interittend long delays retrieving a doc.
Here's the code we're using to retrieve a document from the Centera device:
...
pool = New FPPool(clusterAddress)
Using clipRef = New FPClip(pool, contentAddress, FPMisc.OPEN_ASTREE)
For Each tag As FPTag In clipRef.Tags
Dim tagName = tag.ToString
If tagName.Contains("PageNumber") Then
Dim len = tag.BlobSize
' Retrieve the blob into a buffer
Dim blobSize As Integer = CInt(tag.BlobSize)
Dim converter As New UTF8Encoding()
Dim buffer As Byte() = New Byte(blobSize) {}
Dim source As IntPtr = Marshal.AllocHGlobal(blobSize)
Using streamRef = New FPStream(source, blobSize, FPStream.StreamDirection.OutputFromCentera)
tag.BlobRead(streamRef)
Marshal.Copy(source, buffer, 0, blobSize)
Marshal.FreeHGlobal(source)
streamRef.Close()
End Using
Dim pageNumber = tagName.Replace("PageNumber", "")
myPage = New PageEntity(docId, "Page" & pageNumber, document.DocumentName, buffer)
document.Pages.Add(myPage)
End If
startTime = Now
tag.Close()
tag.Dispose()
Next
clipRef.Close()
End Using
...
I've attached a Centera log file. To generate this file, I made 33 consecutive retrieve requests for 33 separate 2-page documents. There were two docs that experienced a delay:
1. A 19 second delay at 2011-10-26 13:18:02.162. This appears to be occurring in my tag.BlobRead(streamRef) call.
2. An 18 second delay at 2011-10-26 13:18:27.177. This appears to be occurring in my New FPClip(pool, xuid.ContentAddress, FPMisc.OPEN_ASTREE) call.
I can provide a log for a larger sample size if needed. In many cases, the delay is much longer than 19 seconds. These two calls (tag.BlobRead and New FPClip) seem to be the ones that consistently cause the delay.
Thank you for any help you can provide,
Jon
mfh2
208 Posts
0
October 27th, 2011 07:00
Hello Jon -
Nice job on putting together a complete and logically coherent set of facts about this issue. It seems clear to me that the problem in this case is on the Centera end of the wire. There are no socket errors, extra probes or sdk retry logic being triggered in your problem transactions; the Centera is just taking a long time to respond to clip/blob retrieval requests. You should probably open a service request with EMC to have them take a look at the cluster state - save your SDK logs showing the problem as the transaction identifiers can be helpful.
Please note that I am not an EMC employee and these are simply my opinions.
Good Luck,
Mike Horgan
jblank1
3 Posts
0
October 27th, 2011 12:00
Thank you, Mike. You came to the same conclusion that we initially did - that the problem was on the Centera end. However, late yesterday we came across the ClipBufferSize property on the FPPool class, which defaults to 16 KB. We bumped this up to 1 MB and that seems to have resolved our problem. We're now seeing consistently good performance.
Prior to coming to this solution, we had played with setting the FP_OPTION_BUFFERSIZE environment variable on the server (after seeing this thread -> https://community.emc.com/thread/2621), which I believe is the same thing, but it didn't seem to make a difference for us. It wasn't until we set the ClipBufferSize on the pool object directly that we saw an improvement.
Thanks again,
Jon
jblank1
3 Posts
0
October 27th, 2011 13:00
Mike,
That's interesting. While we were still looking for a solution, our infrastructure guys opened a support ticket with EMC. We (us developers) discovered the ClipBufferSize that afternoon, and shortly thereafter they came back to us saying that saw some things going on on the Centera end (including regens) that were likely the cause of our problem that they were working on. As soon as we set the ClipBufferSize, though, the symptoms of our problem went away.
We're still waiting to get back from them an indication that they've finished their work on the Centera device, but in the meantime things seem to be running smoothly as far as we can tell.
Jon
mfh2
208 Posts
0
October 27th, 2011 13:00
Hi Jon -
You're welcome, and I hope that is the end of your troubles. But one of the 2 slow transactions in your SDK log is the retrieval of a blob, and I can't come up with an mental model of how the CDF buffer size could affect this operation so drastically, since at that point the SDK has already read the blob CA from the CDF. Maybe someone else could chime in with a theory.
The only other thing that comes to mind is that there were regens or db rebuilds going on in your cluster for a few days which caused degraded performance and the buffer size setting is a red herring.
Hopefully all is well, and congratulations on your successful SDK spelunking.
Regards,
Mike Horgan