VxRail VCF: lcm-bundle-repo datastore usage on disk

The following alert appears on vCenter:

 lcm-bundle-repo datastore usage on disk 

This datastore is a Network File System (NFS) share from the Software-Defined data center (SDDC) Manager and is mounted as an NFS datastore across the management domain by the name LCM-Bundle-Repo. It is also known as lcmmount_vg-nfs--mount in some versions.

root@sddc-man [ / ]# df -h
Filesystem                          Size  Used Avail Use% Mounted on
devtmpfs                            7.9G     0  7.9G   0% /dev
tmpfs                               7.9G   12K  7.9G   1% /dev/shm
tmpfs                               7.9G  1.1M  7.9G   1% /run
tmpfs                               7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda4                            28G  6.3G   20G  25% /
/dev/sda2                           120M   23M   89M  20% /boot
tmpfs                               7.9G  1.2M  7.9G   1% /tmp
/dev/mapper/data_vg-vcf             181G  1.7G  170G   1% /opt/vmware/vcf
/dev/mapper/lcmmount_vg-nfs--mount  503G  343G  343G  100% /nfs/vmware/vcf/nfs-mount
/dev/mapper/data_vg-vmware           63G  3.3G   56G   6% /var/log/vmware
/dev/mapper/data_vg-data            7.9G  144M  7.3G   2% /data
/dev/loop0                          4.0G  4.0G     0 100% /tmp/mnt7647709551461201106
tmpfs                               1.6G     0  1.6G   0% /run/user/1000
	

The issue occurs when the bundles are left on the datastore and were not removed. This led to the datastore being fully consumed.  

The bundle ID must be known to delete it:

Note: The bundle ID is a 32-character alphanumeric string that is assigned to each bundle. Some people may refer to bundles by the shorter number that is used in the names of the files in the bundle. For our purposes here, we need to know the 32-character bundle ID that the SDDC Manager uses to identify each of the bundles.
 

There are a couple different ways to find the bundle ID of the bundles that have been downloaded to the SDDC Manager. The easiest is to look under the bundle history and look for the name of the bundle you are searching for. For example, the first bundle that is listed in the screenshot below:

• In this screenshot, this bundle is used to update VMware Cloud Foundation to 3.9.0. Click the View Details link, to see more information about that bundle, like this:

Another method to get this information is to use an Application Program Interface (API) call. SSH into the SDDC Manager and run the following command to get a list of all the bundles installed on the SDDC Manager. Replace the information for the host and the admin password:

# curl -k https://sddc-manager.vcf.sddc.local/v1/bundles -u 'admin:VMware123!' -X GET -H 'Accept: application/json' | json_pp
	

When you run this command, you see a JavaScript Object Notation (JSON) formatted output that provides all the bundle information including the ID:

Once you get the bundle ID, SSH to the SDDC Manager and run the following command, replacing the <bundle_id> with the bundle ID you want to remove:

# /opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py <bundle_id>

This command outputs a lot of messages as it removes the bundle. Once completed, either use the API command mentioned earlier, or go back to the SDDC Manager UI and look under the bundle history. You should see that the bundle is no longer listed: