R720xd: Datastore corrupt, H710 did not mark a HDD with 1184 reassigned sectors as faulty

Hi all,

I'm pretty upset by a fatal issue and would like to hear your opinion. I've got 4 R720 servers running with ESXi and around 2 weeks ago I've noticed that a Windows VM (BlueIris CCTV) had a few corrupted recordings which have blocked the auto archiving. I've deleted the affected files and did not think about a hardware problem, especially because my server shows "all green" and did not throw any alarms (I'm also getting any alarms or warnings by iDRAC via eMail to react as soon as possible i.e. by replacing a faulty disk).

Sunday I've decided to swap the USB boot key of the server againt a better one. So I took all VMs and the host to a clean shutdown for the maintenance.

After rebooting the server I found the Windows VM and 4 other Linux VMs not booting up. Windows was even "unrepairable". Even worse, I have 14 days of backups from the VMs and all of them have been broken as well (the corrupted files went into the backups as well).

I've migrated the unaffected VMs to other datastores / hosts, bought 4 new HDDs yesterday and have replaced all disks - because the H710 still shows all drives healthy. The last patrol read had finished Saturday without issues.

After I've migrated back the VMs to the "new" datastore (I still need to set up 2 VMs fully from scratch), I took the removed HDDs one by one for examination to my workstation.

The second one showed horrible SMART data:

More than 1000 (!) reassigned sectors and 9 uncorrectable errors!

It's insane why the H710 did not ring all alarm bells with these SMART values! Instead, it let the disk still run for further days and days. If I'd get an alarm in time, I'd be able to replace the disk ... having still usable backups if the rebuild would fail.

I don't see the problem sitting in front of the server (I'm not aware of an "Ignore SMART" switch) ... or should I?

Thanks for your opinions.

Marco