Essentially FUA bits in the SCB (scsi command block) allows to ask the target NOT to cache data in volatile storage. You (the initiator) can ask your harddrive not to cache your writes because you want them permanently written on the media surface via FUA bits. Since symms and DMXs have a PERSISTENT cache, they simply ignore those 2 bits
Probably some more detail would help - this does seem very unusual - but performance issues usually fall into that category. I can't help on setting the Symm to ignore FUA writes.
1) What sort of host/application is running (UNIX, Windows, etc.) on this environment?
2) How are you measuring these response times - I'm assuming via some sort of host performance utility such as Windows performance counters, UNIX sar, etc.?
3) Assuming you have WLA are you hitting the WP limit on your LUN which is resulting in the miss? If so this could be a cache issue resulting in the misses.
4) When you say in sequence do you mean you are using hypers that are spread across just a few physical disks or that they are spread wide across the box?
5) Is there any sort of SRDF replication in this environment?
Sorry if any of these questions seem rudimentary - just trying to dig into this some more as best I can.
I think you'll need an EMC'er to comment on the FUA SCSI Write behavior.
The only other thing I can think is that maybe you have something odd going on with the statistics. Is the application bursting a very large bunch of writes in a short period of time which might cause the cache misses and the poor response times? Since the interval data is by default 15 minutes from WLA the average might look OK, but you just have some very intense peaks? This is a stretch I know but just trying to think of something...
If you have valid support contract you should be able to log a call and get a performance analysis done on the box. Sounds like you've done a lot of homework on this already so it might be worth the effort to try and get someone officially on the hook to research this.
Ahmmmmm ... Errrr .. We are still thinking about your problem .. hmmmmmm but hopefully we'll give you something soon (I hope)
First of all could you please show us the output of the following command ??
# symcfg -v -sid xxx list
-s-
PS this kind of probs isn't my "favourite" .. performances are always a slippery topic and usually takes a lot of time .. However I'm reading .. and trying to understand what a FUA is
Write hit is 95% with "average write times" of over 85ms. (very bad)
Yes .. a 95% write hit is far from being ideal .. But need to investigate further. Could you please explain which metric are you talking about ?? You picked up a single hyper and plotted the "% write hit" for this specific device, am I correct ??
Is this application doing "Forced Unit Access (FUA)" writes? If so, is there any way for me to tell the Symm to ignore FUA writes?
I think that symm/dmx will ALWAYS ignore the FUA bit in the CDB. But I'm working with ENG to understand how the box will handle eventual FUA writes from the host.
Is this application doing "Forced Unit Access (FUA)" writes? If so, is there any way for me to tellthe Symm to ignore FUA writes?
I think that symm/dmx will ALWAYS ignore the FUA bit in the CDB. But I'm working with ENG to understand how the box will handle eventual FUA writes from the host.
ENG confirmed that Symm/DMX always ignore FUA since it offers a persistent storage in cache, so there is no need to dump the content of the cache on the physical drives. I think we have to work on the issue from a different perspective .. .. Can you show us the output of the following command?
Symmetrix ID: 00018570xxxx Time Zone : Central Standard Time
Product Model : 8830 Symmetrix ID : 00018570xxxx
Microcode Version (Number) : 5568 (15C0AA01) Microcode Date : 05.21.2007
Microcode Patch Date : 05.21.2007 Microcode Patch Level : 68
Cache Size : 32768 (MB) # of Available Cache Slots : 736315 Max # of System Write Pending Slots : 589627 Max # of DA Write Pending Slots : 294813 Max # of Device Write Pending Slots : 5700
Symmetrix Total Operating Time : 634 days, 21:32:07 Symmetrix Power ON Time : Wed Mar 01 12:14:28 2006 Symmetrix Last IPL Time (Cold) : Wed Mar 01 18:35:18 2006 Symmetrix Last Fast IPL Time (Hot) : Thu Nov 08 15:55:33 2007
Host DB Sync Time : Mon Nov 26 00:00:26 2007 Symmetrix CLI (SYMCLI) Version : V6.3.2.0 (Edit Level: 787) Built with SYMAPI Version : V6.3.2.0 (Edit Level: 787) SYMAPI Run Time Version : V6.3.2.0 (Edit Level: 787) SYMAPI Server Version : unavailable
Number of Configured (Sym) Devices : 1806 Number of Visible (Host) Devices : 7 Number of Configured Actual Disks : 384 Number of Configured Hot Spares : 0 Number of Unconfigured Disks : 0 Maximum number of hypers per disk : 64
Number of Powerpath Devices : 0 Powerpath Run Time Version : N/A
SDDF Configuration State : Enabled Configuration Change State : Enabled WORM Configuration Level : N/A WORM Characteristics : N/A
Symmetrix Configuration Checksum : 5F5B20 Switched RDF Configuration State : Disabled Concurrent RDF Configuration State : Disabled Dynamic RDF Configuration State : Disabled Concurrent Dynamic RDF Configuration : N/A RDF Data Mobility Configuration State: Disabled Access Control Configuration State : Disabled Device Masking (VCM) Config State : Mixed VCMdb Access Restricted State : Disabled Multi LRU Device Assignment : BY_NUMBER Disk Group Assignments : In Use Hot Swap Policy : N/A
Please do not post messages with full symm ids .. remove the last 3 digits
Cache Size : 32768 (MB)
Max # of Device Write Pending Slots : 5700
Number of Configured (Sym) Devices : 1806
Number of Configured Actual Disks : 384
It looks like this 8830 is full of drives ... 1806 symmdevs (8 or 9 hypers on every drive if using 2-Way-Mir)... But you you said that your host have 21 meta, 8 members each. Anybody else is working on the same symm ?? It looks like your box is a little low in cache .. With WLA you can look also at the work in the backend ...
I think it's becoming really hard to help you further .. But we'll try
How can I remove the symmid from my symcfg output above?
The server in question only has 21 - 8GB hypers. No Metas. They are using between 1 and 2 GB on heach hyper. (total of 33GB of space utilized out of the 183GB allocated.) Backend does not seem to be heavily utilized. DA utilization is around 20%. FA utilization is around 25%.
Since their I/O load is so low, the only explanation I can find is a FUA write.
No write pending events, yet still have write times into the 80ms range. It's just weird. I have graphs from WLA, but am not for sure how to post images or files to the forum...
Unfortunatly I replied to your post so you can't modify your previous post.. I'll ask our admins to hide your box in previous post.
If your host is the ONLY one working on the symm, it looks quite strange to me that your backend is working at 20% ..
FUA does not work with symms so please forget abou that
It looks like a good candidate for a SR .. I think that our customer service have to dig into this mess .. Unfortunatly I can't either open the SR for you or look at the SR and work on it.. I have to ask "someone" else to jump into and have a look. But first of all we need an open SR. ThX
To follow up with Stefano, please at this point open an SR and request the local Performance Geek (that is what they are called, yes) to work with you on this. S/He can gather data via the Service Processor and along with Host data, better graph and assist in what is or not happening, hot and cold spots, but more specifically in response information.
xe2sdc
4 Operator
•
2.8K Posts
0
November 27th, 2007 08:00
bodnarg
2 Intern
•
385 Posts
0
November 19th, 2007 06:00
1) What sort of host/application is running (UNIX, Windows, etc.) on this environment?
2) How are you measuring these response times - I'm assuming via some sort of host performance utility such as Windows performance counters, UNIX sar, etc.?
3) Assuming you have WLA are you hitting the WP limit on your LUN which is resulting in the miss? If so this could be a cache issue resulting in the misses.
4) When you say in sequence do you mean you are using hypers that are spread across just a few physical disks or that they are spread wide across the box?
5) Is there any sort of SRDF replication in this environment?
Sorry if any of these questions seem rudimentary - just trying to dig into this some more as best I can.
AaronMSTRCRD
26 Posts
0
November 19th, 2007 08:00
2) All stats are from WLA. They are consistent with host stats.
3) There are no write pending events according to WLA.
4) Each hyper is on its own spindle. (spread wide)
5) No SRDF.
AE.
PS. Can't believe Dynamox or Stefano have not responded yet....
bodnarg
2 Intern
•
385 Posts
0
November 19th, 2007 10:00
The only other thing I can think is that maybe you have something odd going on with the statistics. Is the application bursting a very large bunch of writes in a short period of time which might cause the cache misses and the poor response times? Since the interval data is by default 15 minutes from WLA the average might look OK, but you just have some very intense peaks? This is a stretch I know but just trying to think of something...
If you have valid support contract you should be able to log a call and get a performance analysis done on the box. Sounds like you've done a lot of homework on this already so it might be worth the effort to try and get someone officially on the hook to research this.
xe2sdc
4 Operator
•
2.8K Posts
0
November 19th, 2007 23:00
First of all could you please show us the output of the following command ??
# symcfg -v -sid xxx list
-s-
PS this kind of probs isn't my "favourite"
Message was edited by:
Stefano Del Corno
xe2sdc
4 Operator
•
2.8K Posts
0
November 20th, 2007 00:00
85ms. (very bad)
Yes .. a 95% write hit is far from being ideal .. But need to investigate further. Could you please explain which metric are you talking about ?? You picked up a single hyper and plotted the "% write hit" for this specific device, am I correct ??
writes? If so, is there any way for me to tell the
Symm to ignore FUA writes?
I think that symm/dmx will ALWAYS ignore the FUA bit in the CDB. But I'm working with ENG to understand how the box will handle eventual FUA writes from the host.
-s-
Message was edited by:
Stefano Del Corno
xe2sdc
4 Operator
•
2.8K Posts
0
November 21st, 2007 01:00
writes? If so, is there any way for me to tellthe
Symm to ignore FUA writes?
I think that symm/dmx will ALWAYS ignore the FUA bit
in the CDB. But I'm working with ENG to understand
how the box will handle eventual FUA writes from the
host.
ENG confirmed that Symm/DMX always ignore FUA since it offers a persistent storage in cache, so there is no need to dump the content of the cache on the physical drives. I think we have to work on the issue from a different perspective ..
# symcfg -v -sid 123 list
AaronMSTRCRD
26 Posts
0
November 26th, 2007 07:00
Time Zone : Central Standard Time
Product Model : 8830
Symmetrix ID : 00018570xxxx
Microcode Version (Number) : 5568 (15C0AA01)
Microcode Date : 05.21.2007
Microcode Patch Date : 05.21.2007
Microcode Patch Level : 68
Cache Size : 32768 (MB)
# of Available Cache Slots : 736315
Max # of System Write Pending Slots : 589627
Max # of DA Write Pending Slots : 294813
Max # of Device Write Pending Slots : 5700
Symmetrix Total Operating Time : 634 days, 21:32:07
Symmetrix Power ON Time : Wed Mar 01 12:14:28 2006
Symmetrix Last IPL Time (Cold) : Wed Mar 01 18:35:18 2006
Symmetrix Last Fast IPL Time (Hot) : Thu Nov 08 15:55:33 2007
Host DB Sync Time : Mon Nov 26 00:00:26 2007
Symmetrix CLI (SYMCLI) Version : V6.3.2.0 (Edit Level: 787)
Built with SYMAPI Version : V6.3.2.0 (Edit Level: 787)
SYMAPI Run Time Version : V6.3.2.0 (Edit Level: 787)
SYMAPI Server Version : unavailable
Number of Configured (Sym) Devices : 1806
Number of Visible (Host) Devices : 7
Number of Configured Actual Disks : 384
Number of Configured Hot Spares : 0
Number of Unconfigured Disks : 0
Maximum number of hypers per disk : 64
Number of Powerpath Devices : 0
Powerpath Run Time Version : N/A
SDDF Configuration State : Enabled
Configuration Change State : Enabled
WORM Configuration Level : N/A
WORM Characteristics : N/A
Symmetrix Configuration Checksum : 5F5B20
Switched RDF Configuration State : Disabled
Concurrent RDF Configuration State : Disabled
Dynamic RDF Configuration State : Disabled
Concurrent Dynamic RDF Configuration : N/A
RDF Data Mobility Configuration State: Disabled
Access Control Configuration State : Disabled
Device Masking (VCM) Config State : Mixed
VCMdb Access Restricted State : Disabled
Multi LRU Device Assignment : BY_NUMBER
Disk Group Assignments : In Use
Hot Swap Policy : N/A
Parity Raid Configuration : N/A
Raid-5 Configuration : N/A
PAV Mode : NoPAV
PAV Alias Limit : 0
SRDF/A Maximum Host Throttle (Secs) : 0
SRDF/A Maximum Cache Usage (Percent) : 0
xe2sdc
4 Operator
•
2.8K Posts
0
November 26th, 2007 09:00
Please do not post messages with full symm ids .. remove the last 3 digits
It looks like this 8830 is full of drives ... 1806 symmdevs (8 or 9 hypers on every drive if using 2-Way-Mir)... But you you said that your host have 21 meta, 8 members each. Anybody else is working on the same symm ?? It looks like your box is a little low in cache .. With WLA you can look also at the work in the backend ...
I think it's becoming really hard to help you further .. But we'll try
AaronMSTRCRD
26 Posts
0
November 26th, 2007 10:00
The server in question only has 21 - 8GB hypers. No Metas. They are using between 1 and 2 GB on heach hyper. (total of 33GB of space utilized out of the 183GB allocated.) Backend does not seem to be heavily utilized. DA utilization is around 20%. FA utilization is around 25%.
Since their I/O load is so low, the only explanation I can find is a FUA write.
No write pending events, yet still have write times into the 80ms range. It's just weird. I have graphs from WLA, but am not for sure how to post images or files to the forum...
AE.
xe2sdc
4 Operator
•
2.8K Posts
0
November 27th, 2007 01:00
If your host is the ONLY one working on the symm, it looks quite strange to me that your backend is working at 20% ..
FUA does not work with symms so please forget abou that
It looks like a good candidate for a SR .. I think that our customer service have to dig into this mess .. Unfortunatly I can't either open the SR for you or look at the SR and work on it.. I have to ask "someone" else to jump into and have a look. But first of all we need an open SR. ThX
dynamox
9 Legend
•
20.4K Posts
0
November 27th, 2007 07:00
MrTS2Symm
113 Posts
0
November 27th, 2007 07:00
I guess I am set in my...old and new ways.
Guru = Geek.
Ah, sorry. Some terms are interchangeable.
Michael
MrTS2Symm
113 Posts
0
November 27th, 2007 07:00
To follow up with Stefano, please at this point open an SR and request the local Performance Geek (that is what they are called, yes) to work with you on this. S/He can gather data via the Service Processor and along with Host data, better graph and assist in what is or not happening, hot and cold spots, but more specifically in response information.
Michael
xe2sdc
4 Operator
•
2.8K Posts
0
November 27th, 2007 08:00
Did our noise wake you up ??
Message was edited by:
Stefano Del Corno