We run oracle & redbrick file-system based database on AIX LPARs...also.. we run oracle on ASM/raw devices too...

These LPARs are virtuals... in an NPIV configuration.. i.e.  they are virtual HBA's that are mapped to the physical's HBA's (8G) on the VIO server.

My query as I stated in the subject is .. how to identify where the performance bottleneck is in an I/O stack?

More details:

1. The IOSTAT on AIX server reports scaring avg read response times .. we have seen response times as high as 150ms..

2. I know that both disk (queue_depth) and hba/adapters (num_cmd_elems) have limits..

3. Also I am aware that the total I/O response times.. involves time taken to traverse from the host to the disk in the disk subsystem.. and this would involve the cache on CLARiiONs.. too

4. How do I make sure.. that the bad response times.. are because of we reaching the adapter limits.. disk limits.. or due to the cache on the disk subsystem itself?

Please suggest or point me in the right direction....

thanks