VxRail: vSAN Traces Files their Location and Size
Summary: On VxRail ESXi nodes, the /vsantraces partition is a Ramdisk, rather than a physical or vSAN disk partition. In this article, we discuss this partition and some ways to fix the partition's high usage. ...
Instructions
What are the vSAN Trace Files?
vSAN traces are log files used for diagnosing and troubleshooting issues with vSAN. These files can be handy for debugging critical performance issues and data-path-related problems.
vSAN traces are saved to /var/log/vsantraces, by default.
The default maximum file size is 180MB, and there is a rotation of eight files. This means that when the ninth file is created, the oldest file is deleted ensuring that only the most recent eight files are kept.
vSAN also generates urgent traces, which provide details on potentially significant problems. By default, these urgent traces are redirected through the ESXi syslog system. If an external Syslog server is defined, the urgent traces are forwarded to the external collector.
How can I check the vSAN Traces configuration?
We can check the vSAN traces configuration in the file /etc/vmware/vsan/vsantraced.conf.
Access the wanted ESXi host by SSH and type the following command:
cat /etc/vmware/vsan/vsantraced.conf | grep -v -E "#|^$"
These values are not the default values. We already changed the default values:
[root@XXXXXXXX:/vsantraces] cat /etc/vmware/vsan/vsantraced.conf | grep -v -E "#|^$" VSANTRACED_LOG_URGENT_TO_SYSLOG=1 # Enable to send urgent logs to Syslog VSANTRACED_ROTATE_MAX_FILES=10 # Maximum number of trace files VSANTRACED_ROTATE_FILE_SIZE=10 # Maximum size of each trace file VSANTRACED_URGENT_ROTATE_MAX_FILES=10 # Maximum number of urgent files VSANTRACED_URGENT_ROTATE_FILE_SIZE=10 # Maximum size of each urgent file VSANTRACED_LAST_SELECTED_VOLUME="/vsantraces" # Directory to store trace files VSANOBSERVER_MAX_MB_SIZE="10" # Maximum size of observer file
We can get details of the vSAN traces using the command line, using the following command:
esxcli vsan trace get
For example:
[root@XXXXXXXX:/vsantraces] esxcli vsan trace get VSAN Traces Directory: /vsantraces # Directory to store trace files Number Of Files To Rotate: 10 # Maximum number of trace files Maximum Trace File Size: 10 MB # Maximum size of trace files Log Urgent Traces To Syslog: true # Enable to send urgent trace to Syslog
How can I check the usage of those files?
We can use the following command to see the current usage of the vSAN traces. In this case, for instance, the current usage is 28% (considering that the maximum size of this ramdisk for vsantraces is 300M in this example):
[root@host01:/vsantraces] vdf -h | grep -i -E "Ramdisk|vsantraces" Ramdisk Size Used Available Use% Mounted on vsantraces 300M 86M 213M 28% --
I want to continue using the default directory for the vSAN traces but I must limit the maximum size of those files. How can I do that?
So, considering that your partition for the vSAN traces has 300 MB in size and you would like to limit those files to a maximum of 200 MB, the following command can achieve it:
esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10
Explaining the command and its parameters:
esxcli vsan trace set = Main command to configure the behavior of vSAN trace files urgentnumfiles = Maximum number of urgent files urgentsize = Maximum size (MB) of urgent files numfiles = Maximum number of trace files size = Maximum size (MB) of trace files
After applying this command, the expected behavior is the ramdisk for vsantraces to be limited to 200 MB (considering that the maximum value is 300 MB, you are using 200 MB, less than 90%).
If necessary, we can delete older trace files. We can access the files directory and apply a loop command to read files based on a specific pattern and then delete those files. In this case, for instance, we are listing all files that begin with vsanObserver–2024 and delete them (you should adjust this command to match your scenario):
for i in `ls | grep vsanObserver--2024` ; do rm -Rf "$i" ; done
Note: The changes take effect immediately and do not necessarily reboot the ESXi host.
I want to use an external/disk place to store those files. How can I do that?
We can use a different place from /var/log/vsantraces to store the vSAN trace files. It can be elsewhere (a local datastore, a Network File Share (NFS) datastore, and so on). In this case, for instance, we use a local datastore to store these vSAN trace files:
1- The first step here is to create the new directory inside the datastore:
# you can use the command “df -h” to see all available mount points in your ESXi system cd /vmfs/volumes/local-datastore1 mkdir new-vsantraces
In this example, the directory /vmfs/volumes/local-datastore1 is mounted in a higher disk partition than the ramdisk.
2- After creating the directory, set it as the new place to store the vSAN trace files:
esxcli vsan trace set -p /vmfs/volumes/local-datastore1/new-vsantraces/
3- check if the previous command changed the directory for the vSAN trace files:
esxcli vsan trace get
Example:
[root@XXXXXXXX:/vmfs/volumes/local-datastore1] esxcli vsan trace get
VSAN Traces Directory: /vmfs/volumes/local-datastore1/new-vsantraces/
Number Of Files To Rotate: 10
Maximum Trace File Size: 10 MB
Log Urgent Traces To Syslog: true
4- This change is applied automatically and does not need to restart any service. We can access the new directory and list the content, we already see some files here:
[root@XXXXXXXX:/vmfs/volumes/local-datastore1] cd new-vsantraces [root@XXXXXXXX:/vmfs/volumes/local-datastore1/new-vsantraces] pwd /vmfs/volumes/local-datastore1/new-vsantraces [root@XXXXXXXX:/vmfs/volumes/local-datastore1/new-vsantraces] ls -l total 1216 -rw-r--r-- 1 root root 43130 Feb 29 14:40 vsanObserver--2024-02-29T14h22m01s.gz -rw-r--r-- 1 root root 950256 Feb 29 14:41 vsantraces--2024-02-29T14h38m21s662.gz -rw-r--r-- 1 root root 24 Feb 29 14:38 vsantraces.index -rw-r--r-- 1 root root 15 Feb 29 14:39 vsantracesDOMObj--2024-02-29T14h38m21s900.gz -rw-r--r-- 1 root root 24 Feb 29 14:38 vsantracesDOMObj.index -rw-r--r-- 1 root root 38611 Feb 29 14:40 vsantracesIODiag--2024-02-29T14h38m21s775.gz -rw-r--r-- 1 root root 24 Feb 29 14:38 vsantracesIODiag.index -rw-r--r-- 1 root root 398 Feb 29 14:40 vsantracesLSOM--2024-02-29T14h38m21s821.gz -rw-r--r-- 1 root root 24 Feb 29 14:38 vsantracesLSOM.index -rw-r--r-- 1 root root 15 Feb 29 14:39 vsantracesLSOMVerbose--2024-02-29T14h38m21s864.gz -rw-r--r-- 1 root root 24 Feb 29 14:38 vsantracesLSOMVerbose.index -rw-r--r-- 1 root root 17953 Feb 29 14:40 vsantracesUrgent--2024-02-29T14h38m21s725.gz -rw-r--r-- 1 root root 24 Feb 29 14:38 vsantracesUrgent.index
Additional Information
For more details on that, check the following VMware KB:
https://kb.vmware.com/s/article/2147956?lang=en_US (External Link)