SC5020 data corruption

First, the issue you point out for Dell Compression was resolved in SCOS07.03.04 so it was not the cause of your issue. Second, what type of SQL are you using? Microsoft, Sybase, MySQL, Oracle... Third, are you running database compression in addition to Data Reduction on the SAN?

Yes, this is something different than the known date reduction issue. I am just curius if anyone else has had issues with corrupted data.

 

Below a bit more detail if the issue we have:

After upgrading SCOS to version 7.3.2.48 to 7.3.5.8 and activating data reduction on volumes we where alerted within a few weeks about database corruption on one MS SQL database. The issue affected older data ( reports ) , new data inserts into database worked ok.

All the volumes we use are live volumes between two SC 5020 units. I had only updated one of the units to 7.3.5.8 while the other was running 7.3.2.48  without data reduction.

During troubleshooting we found that moving the primary volume to the node with 7.3.2.48 and limiting paths to that node resolved the database corruption issue. When we moved the volume back the database broke again. Everything while the vm was up and running.

I did some further testing and shut down the problematic vms. I then took snapshots of the live volume on both units and mounted them to vmware.  I then did a binary comparison on the problem vmdk-flat files.  They should be identical since the vm was shut down during snapshot.  Binary comparison gave that 216KB were different in the 260GB flat-vmdk file holding the database. That explained why IO served from 7.3.5.8 behaved differently than data served from the 7.3.2.48 unit. Recovering is simple as we have good data on the 7.3.2.48. The issue is not ms sql specific, but if the incorrect bytes hits your db you will now about it.

It would seem that issue is extremely rare since we seem alone with this. Dell techs are doing a good job trying to figure out what happened, I just wanted to share what we found and check if any of you have seen anything similar. 

 

 

 

 

 

Hi, There has been no progress on the issue and the latest suggestion from dell support is all but helpful: 'There is nothing more we can do here'. Since i did find corruption on some luns my gameplan is to delete all the luns on the problematic system. Recreate the live volumes from the healthy node and then activate data reduction. Do some database checks and some random binary vmdk comparisons. Perhaps then upgrade the other system, but probably never dare to activate data reduction on that one.