PowerFlex: CPU Concepts And Troubleshooting

Concepts Explained

• Central Processing Unit (CPU) - The CPU is a piece of hardware that carries out the instructions of a computer program.
  It performs the basic arithmetical, logical, and input/output operations of a computer system.
  Typically seen as the processor or socket

• CPU Core - A CPU core is the actual CPU processing component.
  It handles all command sequences received from a thread.
  Also seen as pCPU (VMware)

• CPU Thread - A CPU thread is a virtual version of a CPU Core.
  A thread is merely an ordered sequence of instructions.

• Logical Cores - Each CPU thread is compared directly to a Logical Core. Logical Cores measure the number of operations a physical core can run simultaneously.
  With Hyper-Threading capability, one physical core can run two operations and therefore have two logical cores.

• Hyper-Threading - This is a process where a CPU splits each of its Physical Cores into Virtual Cores, which are known as Threads.
  Can be enabled/disabled on an operating system (OS) and BIOS that support it.

• Virtual Central Processing Unit (vCPU) - Each vCPU is seen as a single physical CPU core by the virtual machine's (VM) OS.
  Represents a portion or share of a Physical Core that is assigned to a VM
  Depending on the Hypervisor, each Physical Core would support 5-10 vCPUs per VM.

CPU > Core > vCPU diagram

Physical Core vs. Logical Core
lscpu command output

lscpu | egrep 'Socket\(s\)|Core\(s\) per socket|Thread\(s\) per core|CPU\(s\)\:' | tac

Example output:

Socket(s):              2    ← The motherboard has 2 CPU processors/sockets.
Core(s) per socket:     8    ← Each CPU socket has 8 physical cores. Hence, it has 16 physical cores in total.  
Thread(s) per core:     2    ← Each physical CPU core can run 2 threads = Hyper-Threading enabled.
CPU(s):                 32   ← The computer has 2 × 8 × 2 = 32 logical cores in total.

The total index of logical cores = CPU sockets × physical cores per socket × threads per physical core.

/proc/cpuinfo file output

echo " "; echo "Model Name: "$(cat /proc/cpuinfo|egrep 'model name'|head -1|awk '{for(i=4;i<=20;i++)printf "%s ",$i;print ""}'); cat /proc/cpuinfo | egrep 'processor|physical id|siblings|core id|cpu cores' | sed "s/^processor/\n&/"

Example output:
All the output values are zero-based.

processor     : 0    ← The index number of threads.
physical id   : 0    ← CPU socket index #0 (the only one if no other physical id's in the full output).  
siblings      : 8    ← The amount of logical cores in the socket.
core id       : 0    ← CPU core index #0.
cpu cores     : 4    ← CPU socket index #0 has 4 physical cores.

The total number of logical cores = siblings × total amount of physical id's

PowerFlex Documentation and Logs

To find CPU-related information about the PowerFlex server, you can use the following methods:

On live system ONLY
lscpu

echo " ";echo "Model Name: "$(lscpu|egrep 'Model name'|awk '{for(i=4;i<=20;i++)printf "%s ",$i;print ""}');echo " ";lscpu|egrep 'Socket\(s\)|Core\(s\) per socket|Thread\(s\) per core|CPU\(s\)\:'|tac

Example output:

Model name:             Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz

Socket(s):              1
Core(s) per socket:     4
Thread(s) per core:     2
CPU(s):                 8

On live system AND getInfoDump
/proc/cpuinfo

echo " ";echo "Model Name: "$(cat /proc/cpuinfo|egrep 'model name'|head -1|awk '{for(i=4;i<=20;i++)printf "%s ",$i;print ""}');cat /proc/cpuinfo|egrep 'processor|physical id|siblings|core id|cpu cores'|sed "s/^processor/\n&/"

For getInfoDump you must change the cpuinfo file location, it is located under ...\getInfoDump\server\proc_cpuinfo.txt.

The /proc/cpuinfo file is more comprehensive than lscpu - for additional information use the following LINK.

Example output:

Model Name: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz

processor       : 0
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4

processor       : 1
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4

processor       : 2
physical id     : 0
siblings        : 8
core id         : 1
cpu cores       : 4

processor       : 3
physical id     : 0
siblings        : 8
core id         : 1
cpu cores       : 4

processor       : 4
physical id     : 0
siblings        : 8
core id         : 2
cpu cores       : 4

processor       : 5
physical id     : 0
siblings        : 8
core id         : 2
cpu cores       : 4

processor       : 6
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4

processor       : 7
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4

For PowerFlex System requirements and recommendations, see the official documentation on the Dell Support website.

Troubleshooting Tools and Tips

Linux
top
This tool gives a readout of users, tasks, CPU load, and memory usage.
To quit the top function, press the letter 'q' on your keyboard.

Some standard command switches:
-i
Idle-processes
This hides all the idle processes, making it easier to sort through the list.

-n
Number-of-iterations
Specifies the maximum number of iterations, or frames, top should produce before ending

-b
Batch-mode
Starts top in Batch mode, which could be useful for sending output from top to other programs or a file

Some other useful commands while top is running include:
M - sort task list by memory usage
P - sort task list by processor usage
N - sort task list by process ID
T - sort task list by run time

mpstat
This tool is part of a software bundle called sysstat, which displays CPU Activity. The system displays usage for each processor (or processor core).

%usr – % CPU usage at the user level.
%nice – % CPU usage for user processes labeled “nice”.
%sys – % CPU usage at the system (Linux kernel) level.
%iowait – % CPU usage idling waiting on a disk read/write.
%irq – % CPU usage handling hardware interrupts.
%soft – % CPU usage handing software interrupts.
%steal – % CPU usage being forced to wait for a hypervisor handling other virtual processors.
%guest – % CPU usage spent running a virtual processor.
%idle – % CPU usage on idle time (no processes, and not waiting on a disk read/write).

iostat
Command for Average Usage. The system displays average CPU usage since the last boot.

sar
This tool manages system resources.

Common command switch:
-u
This tracks CPU performance only

%user: Percentage of CPU utilization that occurred while executing at the user level (application).
%nice: Percentage of CPU utilization that occurred while executing at the user level with nice priority.
%system: Percentage of CPU utilization that occurred while executing at the system level (kernel).
%iowait: Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
%idle: Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

The following command displays the top 10 CPU users on the Linux system:

ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

OR

ps -eo pcpu,pid,user,args | sort -r -k1 | less

VMware
In virtualization environments, especially the commonly used VMware ESXi, the use of system resources is layered and sometimes too comprehensive to analyze quickly.
Follow this VMware KB for various code versions in troubleshooting ESXi host/VM performance issues - Troubleshooting VMware Resources.

Both UI and CLI can be used for troubleshooting CPU and other resources' performance.

UI:
vCenter Server collects statistical data from all ESXi hosts and VMs connected and aggregates, calculates, and archives the data in configured intervals.
To access the data, start vSphere Client, select your VM or host, and go to Monitor > Performance > Overview to get a summary of main indicators, like CPU, Memory, Disk, and Network.
Go to Advanced and use the View drop-down menu at the upper right corner to see more details about each indicator.

CLI:
esxtop
This tool gives a readout of users, tasks, CPU load, and memory usage.
Log in to the host SSH session (root privileges required) and type esxtop. When you start ESXTOP, you get the CPU view by default.
See the following article for a more comprehensive understanding of the various esxtop metrics and screens - ESXTOP Explained.

Notice that for a specific OS/ application performance on a VM, follows the relevant instructions provided by the relevant vendor, per OS/application.

Some standard command switches:
-l
In large environments, esxtop can have high CPU utilization due to the amount of data that must be gathered and calculations that must be done.
Locks specific entities and keeps esxtop from gathering detailed info to limit the amount of CPU power needed

-u
This option suppresses the server's physical CPU stats, which save a significant amount of data being stored.

-n
Number-of-iterations
Specifies the maximum number of iterations, or frames, esxtop should produce before ending

-b
Batch-mode
Starts top in Batch mode, which could be useful for sending output from top to other programs or a file

Windows
In Windows environments, high usage of CPU and memory is not always an issue that must be fixed. It would vary according to use cases.
We can use Task Manager and see which application/task/service causes high usage of resources.

Since Windows servers can be used for various purposes, running multiple services.
Follow this Microsoft KB for troubleshooting performance issues for various services - Troubleshooting Windows Server Components.

PowerShell
A cross-platform task automation and configuration management framework, consisting of a command-line shell and scripting language
See the following article for a more comprehensive understanding of CPU troubleshooting using PowerShell - PowerShell CPU Troubleshooting.

References

• https://www.accuwebhosting.com/blog/what-is-virtual-processor-or-vcpu/

• https://en.wikipedia.org/wiki/Central_processing_unit