SCSI reservations

Did emc support mention that there was a persistent reserve on that disk? If yes was the device enabled with SCSI3_Persist_reserv flag? I haven`t come across a scenario where we have this flag set for a standalone Windows 2003 host. kindly refer emc87610 for more details about the environment where the flag is needed.

I would not recommend to setup this flag if you do not need it.By enabling PER on devices that do not really need it, you face the slim possibility of having I/Os fail to hosts (during drive problems) that could otherwise have been completed successfully.

Cheers

Andy.

Hi Andy,

Yes, EMC support mentioned that there was a persistent reserve on the disk (which is consistent with the fact that "symdev -sid xxx list -resv" didn't show anything -- I understand that SE doesn't show persistent reserves). However, the SCSI3_persist_reserv flag was *not* enabled, so this is the inconsistent bit. *None* of the devices in our 18 or so symms have this flag enabled.

Regards,

Jurjen

How did you get this issue fixed?? Did you reboot the Servers or the Symmetrix Hardware team cleared the Persistent reservation from the devices? I really doubt that in Server 2003 without PER bit enabled, it saves persistent reservation information on the Devices?

Regards,

Saurav

Well, how the device became accessible again is a bit unclear, unfortunately.

I allocated the device to several Windows-based hosts, but all had the same symptom. Then, I allocated the device to an AIX server. The device wasn't accessible to AIX as well, with the same error (resource in use). However, the first time someone looked at the device after that there suddenly was a Group reservation on the device. The first person looking happened to be an EMC support engineer, and he cleared the reservation. Unfortunately, I don't know over which FA that reservation was set. When I looked somewhat later there was a new Group reservation on the device (set over the FA over which the AIX machine was attached). This surprised me, because the device was no longer visible to that AIX host, however, I'm not 100% sure when this reservation was placed, so I'm assuming it was set by the AIX host right before I made the device invisible to it. Anyway, clearing that second Group reservation was simple, and after that the device worked just fine, for every host I tried so far.

This is all still somewhat strange: placing a Group reservation couldn't have been possible with a persistent reservation already on the device. At the same time, it also couldn't have been possible for a persistent reservation to be on the device (because the PER-flag wasn't enabled).

My guess (without the benefit of knowing the inner workings of Enginuity ): we somehow happened to hit a peculiarity in how Enginuity handled the reservations for this device.

Regards,

Jurjen

Hi there,

If anybody happens to know that the SCSI3_persist_reserv flag (also known as the PER flag) actually controls: I'm still a bit curious.

From all I've read I think the flags controls whether or not the symm accepts the option for a reservation placed by PERSISTENT RESERVE OUT command to survive a power cycle, but I haven't yet been able to confirm this.

Thanks,

Jurjen Oskam

You can enable PER flag on selected volumes. When you enable the PER flag the volume will accept SCSI3 Persistent Reservation commands and behave as stated in SPC-3 standards.

A volume will accept the PERSISTENT RESERVE OUT command even *without* the PER flag (see my first post in this thread): this results in what Solutions Enabler calls a "Group" reservation. The PERSISTENT RESERVE OUT command has an optional field: the host can optionally request for a reservation to persist even across a power cycle of the storage. I believe (but have not seen it confirmed) that the PER flag controls this optional feature: without the PER flag, the symm will not accept the request of a host for a reservation to persist across a power cycle of the storage. With the PER flag, the symm will accept such a request...

In my experience many clustering software (remarkably TSFKA Veritas Cluster Service) will simply complain and quit if the devices don't have the required PER bit enabled. This behaviour is quite different from what you described.

PER bit enables SCSI3 PGR on selected devices.

http://storagenerve.com/tag/scsi-3-pgr-reservation/

First off: I'm no longer looking for the cause of the host crash and subsequent inaccessible device. It looks like we've hit some rare set of circumstances which triggered an obscure and rare bug in Enginuity. Unless someone can convince me otherwise, that's my explanation for what happened.

Now I'm just interested in what the PER flag actually influences. The thing is, everybody mentions things like "SCSI-3 reservations", "SCSI3 PGR", etc. When reading the actual SCSI standards (SPC-3), there are only three commands that can be used to place a reservation on a device (with the first two being deprecated, even):

* RESERVE(6)

* RESERVE(10)

* PERSISTENT RESERVE OUT

The first two only deal with single-inititiator reservations. Solutions Enabler calls these "Exclusive". Exclusive reservations are always possible: the value of the PER flag doesn't matter. Looking at e.g. emc212199, I know that a volume will also accept the PERSISTENT RESERVE OUT command, even while the PER flag is not set.

So, having concluded that any volume will accept all three commands in all cases, irrespective of the PER flag being enabled or not, this lead me to the question: if all three commands are accepted no matter the value of the PER flag, what does the PER flag actually influence? Which behaviour is changed by setting or clearing the PER flag on a device?

Further reading the standard, I noticed that there's an optional feature with respect to reservations placed by the PERSISTENT RESERVE OUT command. That optional feature is the ability for a reservation to survive a power cycle of the storage (which is not the case by default). When the storage offers this optional feature, it sets a flag in the INQUIRY data so that the host can tell that a particular volume (LU) offers this optional feature. When placing a reservation on that LU, the host can then choose for the reservation to persist across power cycles of the storage.

I didn't find any other optional features regarding reservations in the SCSI standards. So, the most obvious guess is that the PER flag controls the optional feature to offer the possibility of a reservation to persist across a power cycle. With the PER flag enabled, the volume offers that optional feature, without the PER flag, it doesn't.

Anyway, I think I'll try to find a Linux box, attach it to a symm and experiment with the sg_persist command, in particular the "--param-aptpl" option. This allows me to play with these reservations myself.

I checked with a SME on this issue and received the following:

The SCSI3 persistent reservation (PER) flag is set at the Symmetrix volume level. It is not a director flag setting and should not be confused with the SCSI3 director flag.  It allows the volume to interpret the SCSI-3 Persistent Reservation commands, 0x5E and 0x5F.

When the PER bit for a device is set, the persistent reservation information is stored on the CE cylinder of the devices that are configured for this support.  If an I/O goes to one of these devices, that access privilege of the host that generated the I/O against this information must first be checked.

Refer to EMC Knowledgebase solution "emc87610."

The PER bit is normally used in Sun Cluster 3.0 environments, where two or more channels access the device. SCSI-3 PERs are used extensively by clusters for protection and data safety.

Well, I got around to do some experimenting. My experiments were done on a DMX-3 running 5773.160.111, on a device that was accessed over a director with SC3 and SPC2 enabled, and OS2007 disabled. No port flag overrides. I used a Windows virtual machine (on ESX 3.5.x), with the device in question made available to Windows in the form of a VMware Raw Device Mapping (i.e.: pass-through of raw SCSI commands). To send the SCSI commands, I used the Windows build of the sg3_utils package.

With the SCSI3_persist_reserv flag disabled, there was no way to register a reservation key with the PERSISTENT RESERVE OUT command. All my attempts to register a reservation key were unsuccesful. Since a reservation key is required to place any type of persistent reservation, this effectively means that it's not possible to use PERSISTENT RESERVE OUT to place any kind of reservation on a device when the SCSI3_persist_reserv flag is disabled. The optional feature of being able to persist through a power loss is not relevant. In short: persistent reservations are not possible, as people already mentioned in this thread.

BUT: it's documented that PowerPath on at least AIX does use the PERSISTENT RESERVE OUT command: it uses this command to place Group-reservations on a device. This works, even though the SCSI3_persist_reserv flag is disabled. So, this must mean that PowerPath uses some sort of "secret handshake" with the symm, so that the symm will let PowerPath use PERSISTENT RESERVE OUT command wile denying this command to everybody else. My guess would be that PowerPath uses a "magic" reservation key of some sort, which it doesn't need to register first because it's built into Enginuity. But since that's just a guess, I'll head over to the PowerPath forum and ask around there.

With the SCSI3_persist_flag enabled, registering a reservation key and placing reservations worked as expected, with two exceptions. There are six types of persistent reservations, and four of the six types worked as the SCSI standard describes. Two of them don't, though: reservation type 7 (Write Exclusive - All Registrants) and type 8 (Exclusive Access - All Registrants). Normally, when releasing a reservation, you'll need to provide the reservation key that was used to place the reservation. For reservation types 7 and 8 this failed: releasing those reservations did not work. It did work when using a reservation key of "0". This seems incorrect behaviour. (It's certainly against SPC-3 and the current draft of SPC-4.) Since we're not affected by this in any way (we do not use persistent reservations anywhere), it doesn't really make sense to open a Service Request. I have a detailed log and description available (including the exact SCSI commands and parameters sent, and the results), so if anyone is interested just let me know.