VxRail: Troubleshooting Issues with VxVerify Prior to a VxRail Upgrade

This KB article is dedicated to troubleshooting issues that prevent VxVerify from successfully running.

This knowledge article is referenced from tests that do not have an article targeted for their relevant response (for example: Warning, Failure, and Critical). An example of this would be a query that returns an unexpected response from a query.

VxVerify is designed to detect issues which could cause complications or failures during VxRail upgrades. VxVerify creates Python programs that are known as Minions, which are sent to the VxRail nodes. Below are the typical test results that should be expected in vxverify_tests.json:

If any false positive test result is discovered, gather the VxVerify logs and engage Dell Support to open a Jira VXV ticket with VxRail Engineering.

There are multiple causes which can prevent VxVerify from running successfully.

• The most common cause of failures is the Python script having expired. Each VxVerify version is set to only last for two weeks from the date that it was published. This does not apply when VxVerify is run as a plug-in to the VxRail healthcheck framework (which changes the VxVerify functionality).
• Other reasons can be permissions issues on the VxRail manager or communication issues with the hosts.
• If the cause of the event is unclear, engage Dell Support to open a VXV ticket with VxRail Engineering.

The below section provides instructions on how to collect logs and troubleshoot if VxVerify is not running properly.

Log Collection for Support Engagement

When engaging support with VxVerify related issues, upload the entire archived vxv folder for analysis, or the vxverify log .zip file. Current stand-alone VxVerify versions save an archived file to /tmp, which has all the necessary results and logs for analysis.

For example: /tmp/vxverify-c9.zip

In addition to this .zip file from the most recent run of VxVerify, up to five sets of previous logs may also be present in the /tmp folder. The names have the date-time at which these were run in the file attributes: 

vxv_previous_01.zip

Alternatively, run the below command to archive all the relevant files:

tar cvzf vxverify-2020-12-31.tgz /tmp/vxv/

Troubleshooting

Typical troubleshooting steps for running VxVerify 2 (for VxRail 4.5, 4.7, and 7.0.000)

• VxVerify needs Python to run and the command line to run it is as follows:

python /tmp/vxv/vxverify.pyc

• If a magic number error occurs, this usually means that the wrong version of VxVerify is being used for the Python version on VxRM. For example, the below error is encountered if you run VxVerify 2 on 7.0.320.

RuntimeError: Bad magic number in .pyc file

• To check that the correct version of VxVerify is being used, see article ( link below requires authentication to Dell Support portal):
  • VxRail: How to run the VxRail Verify tool (VxVerify Releases section) 
• VxVerify creates programs that are called minions, which are sent, run, and are retrieved using SSH. If SSH can be used, even if it is not already enabled, VxVerify turns on SSH for each host for the commands to be run. If the hosts are locked down where SSH cannot run, the minions cannot run, and the host tests return a result code: 2 (Failure) or 3 (Critical). If this happens, discuss the SSH permissions with the administrator.
• VxVerify2 is designed to run on Python 2.7, which should be present on the VxRM VM and is usually the only Python version available. If there are more Python versions, run the following to test it (this has the -h option, which is --help):

python2.7 vxverify.pyc -h

VxVerify writes logs and output files to /tmp/vxv/. If it does not have sufficient permissions, it cannot run. Check the folder permissions with the following commands. If there are no read/write permissions for all users, add these with chmod (root permissions may be required):

$ ls -lad /tmp/vxv

drwxrwxrwx  3 mystic   users      4096 Oct  1 07:42 vxv

• If permissions errors persist with VxVerify (for example: 'Permission denied for deleting previous logs.'), try deleting the vxv folder with the following command (the root password is needed for sudo access). After running this delete command, VxVerify must be installed again using only mystic permissions:

sudo rm -r -d /tmp/vxv

• Another option is to save the VxVerify output files to a new folder. VxVerify creates the tree if it does not exist using the -l or --log option, followed by the path that the logs should be saved to. For example:

python vxverify.pyc -l /tmp/vx1

Troubleshooting Differences between VxVerify2 and VxVerify 3 (for VxRail 7.0.010+)

For VxRail 7.0.010 and later, VxVerify 3 must be used, due to fundamental changes in VxRM.

The same troubleshooting steps for VxVerify2 also apply to VxVerify3, EXCEPT:

• The version of Python for VxVerify3 is 3.6.
• The location of the site packages that VxVerify requires changes, which sit in one of the locations below:
  • /mystic/telemetry/DCManager/venv/lib/python3.6/site-packages
  • /mystic/radar/venv/lib/python3.6/site-packages
• If both of these folders are inaccessible from the mystic user, the VxVerify program gives an error message about the site packages and exit.
  • A workaround is using the root user.

Timeouts

If the minion takes longer than 20 minutes to complete, VxVerify must supply a timeout event in the summary table.

| Node-name  |Critical 66460| minion: Maximum run time for minion exceeded

• The corresponding vxv.log entries for this are:

2022-10-06 06:42:51-WARNING  Creating a fault json for missing minion

2022-10-06 06:42:51-WARNING  [fail_minion] Producing result file for lab-esx22, due to: Maximum run time for minion exceeded. See minion logs in /tmp/vxv

• This should be checked by looking at the minion log for that host to see if the minion encountered an error that caused it to stop or whether the testing was continuing slowly and it ran out of time to complete.
• If a minion does complete correctly, the following should be the last lines in the log:

2022-10-06 06:39:30 INFO    Writing to JSON: /tmp/xc882-61f1f3b9-7cb2-130a-35b0-minion.json

2022-10-06 06:39:30 INFO    JSON save: os.stat_result(st_mode=33206, st_ino=47144, st_dev=1, st_nlink=1, st_uid=0, st_gid=0, st_size=499562, st_atime=1665038370, st_mtime=1665038370, st_ctime=1665038370)

2022-10-06 06:39:30 DEBUG   ESXi minion's work is done.

JSON file transfer not working

If the host minion results cannot be transferred using SCP or SFTP back to VxRM, the following may be seen on the output table:

| Node-name  |Critical 66460| minion: No JSON downloaded from node via SSH

• The corresponding vxv.log entries for this are:

2022-10-06 06:42:51-WARNING  Creating a fault json for missing minion

2022-10-06 06:42:51-WARNING  [fail_minion] Producing result file for lab-esx22, due to: No JSON downloaded from node via SSH. See minion logs in /tmp/vxv

• This should be checked by looking at the minion log for that host to see if the minion encountered an error that caused it to stop or whether a JSON file was produced, but could not be accessed using SSH.
• If a minion does complete correctly, the following should be the last lines in the log, which shows that a JSON was successfully saved into the /tmp folder of the node:

2022-10-06 06:39:30 INFO    Writing to JSON: /tmp/xc882-61f1f3b9-7cb2-130a-35b0-minion.json

2022-10-06 06:39:30 INFO    JSON save: os.stat_result(st_mode=33206, st_ino=47144, st_dev=1, st_nlink=1, st_uid=0, st_gid=0, st_size=499562, st_atime=1665038370, st_mtime=1665038370, st_ctime=1665038370)

2022-10-06 06:39:30 DEBUG   ESXi minion's work is done.

The recommended steps to troubleshoot the above would be:

• If using VxVerify3, try running with the --fix flag, which uses an alternative SSH mechanism. This can avoid some SSH permissions problems.
• Specify a new path using -1 (the folder does not need to be created first), which can help if VxRM permission errors are preventing the file transfer:
• For example, > python vxv2.pyc -l \tmp\vxv0
• If the above still does not work, take a snapshot of VxRM, before cleaning up any unnecessary files and folders in /tmp and /home/mystic, such as those previously used by VxVerify.
• Restart VxRM (VxRail Manager).

If VxVerify encounters an error that cannot be fixed with the above steps, save the vxv log bundle as detailed above and escalate the issue with Dell Support.

Login and credential errors in VxVerify2 and VxVerify3

• To run tests on VxRM (VxRail Manager) and the VxRail nodes, VxVerify does not require any credentials. VxVerify can access the encrypted credentials directly from the VxRM database and decrypts these before using them. Sometimes the VC management credentials cannot be accessed, in which case these credentials can be specified in the VxVerify command line:

python vxverify.py --verbose -u vxrailmgmt@localos -p ChangeMe1!

• The use of some special characters that are permitted by vCenter, can cause issues for VxRail Manager (particularly for any commands that utilize the Linux Shell). Check that none of the following characters are used in the passwords for vCenter or the ESXi hosts:

` $ % / \

• If in doubt as to which username is being used for management, consult the vxv.log. For example:

... - DEBUG Users from runtime & settings records:     vxrailmgmt@localos & vxrailmgmt@localos

• Tests to vCenter using SSH require a root username and password, which can be specified with the -r and -w options respectively. The vCenter tests are only run if these are specified (although if the root user is root, which is the default, then only the root password must be specified). For example:

python vxverify.py --verbose -w R00tPassword!

Critical test failures examples

These can be the result of severe errors, which prevent further tests from being run. For instance, if the vCenter management username and password that is saved in VxRM is not current, then all queries to the VC API fail and no further testing can be run. Examples of this include:

#========================#======#=========#====================================================================#==============#
|  Hostname / Category   |Status  Dell_KB |  Warnings or Failures, unless tests Passed                         | Product S.N. |
#========================#======#=========#====================================================================#==============#
| VxRM                   | Critical 66460 | vc_external: VC MOB API connection failed                                        .| <- Stored vCenter password rejected
| VxRail                 | Critical 66460 | vxtii_err: Internal DO host query failed. See vxv.log for details in /tmp/vxv    .| <- ESXi node data cannot be retrieved from the VxRM config service
| _cluster               | Critical 66460 | esx_vers: No valid ESXi test results found                                       .| <- No ESXi tests could be run

VxVerify files

VxVerify creates the following files in /tmp/vxv/ or /var/log/mystic/vxv/ (unless another logging folder is specified using the -l argument).  These files are all saved in a single archive file in /tmp, such as /tmp/vxverify-569ae010.zip or /tmp/vxv_previous_01.zip.
Checking these files manually may help to find issues on the cluster even if the VxVerify script does not fully complete:   

• vxv.log - (log file for the VxVerify script)
• minion_hostname.log - (remote log for the minion script running on each host)
• minion_hostname.txt - (remote text output for each minion, showing which test number is in progress)
• /json/host_uid.json - (file produced by each minion, with the test results, which is later merged with other host data, then deleted)
• vxverify_tests.json - (combined output for all tests, which can be checked manually to see every test result)
• vxtii.json - (combined responses from hosts for queries such as the iDRAC Hardware Inventory)
• vxtii.txt - (report summarizing iDRAC and ESXi information for each node)
• vxverify.txt - (the summary table, which is also shown on the screen, if VxVerify is not run in quiet mode)
• vxverify.html - (combined VxVerify and VxTii reports in HTML format) (only present when running VxVerify directly, rather than embedded in VxRail Manager healthchecks)