Highlighted
WillSanders
2 Iron

InsightIQ Local /datastore Expansion

Hello,

Working on a new implementation of InsightIQ.  The default /datastore size

created by the OVA is 6.5GB, and we'd like to expand this.  How do we expand

the local /datastore file system?  We've already expanded the ESX datastore

that supports the InsightIQ VM.

Please note that we do NOT want to use NFS for /datastore.  We want to

expand the local /datastore file system.

Thanks in advance,

Will S.


Labels (1)
8 Replies
johnsonka
3 Argentium

Re: InsightIQ Local /datastore Expansion

Hello WillSanders,

Thank you for your question! To expand the datastore partition on InsightIQ, you will need to create a new virtual disk and move the data (if there is any.) If you have a login to EMC online support, you can follow the resolution in this Knowledge Base article:

https://support.emc.com/kb/167800

If you do not have a login to EMC online support, send me a message and I can get the information you need. Please let me know if there is anything else I can do for you.

WillSanders
2 Iron

Re: InsightIQ Local /datastore Expansion

Katie,

Thanks for the response.  I do have access to support.emc.com, but following the link I get an "Access Denied" error.

Can you forward me the information?

Thanks,

Will


0 Kudos
dynamox
6 Thallium

Re: InsightIQ Local /datastore Expansion

Article Content

Issue

Statistics from Isilon clusters are no longer being gathered. This issue applies to all virtual machine (VM) installations of InsightIQ.

NOTE
This article applies only to Virtual Machines provided by Isilon. For physical machines, please refer to your system documentation about how to add devices.

Symptoms

The /var/log/insightiq.log file on the InsightIQ VM contains entries similar to the following:

2013-07-18 12:59:54,413 ERROR [insightiq.lib.cluster.errors] mycluster Cannot recover from exception - monitoring will not restart: Traceback (most recent call last):
File "/usr/share/isilon/lib/python2.6/site-packages/insightiq/lib/cluster/monitor.py", line 44, in _run
data = self.service()
File "/usr/share/isilon/lib/python2.6/site-packages/insightiq/lib/cluster/fact/pruning_monitor.py", line 75, in service
raise DatastoreDiskFullError("Data Store Disk Full")
DatastoreDiskFullError: Data Store Disk Full
2013-07-18 12:59:54,406 INFO  [insightiq.lib.cluster.fact.pruning_monitor] Data store /net/smartconnect.mycluster.com/ifs/InsightIQDataSt
ore/3_0007430779ddd058fc4f7b0f27ba92c92d3c_1374168109 free space is less than 1 GB, however, there are no available chunks to delete.

Isilon
Isilon OneFS
Isilon InsightIQ

The amount of data being gathered for InsightIQ exceeds the allotted datastore size.

NOTE
By default, the InsightIQ datastore is 64 GB in size.

Solution

Perform the following steps to create a new, larger local datastore, configure InsightIQ to use the new datastore, and copy the existing data from the old datastore to the new datastore.

  1. Open an SSH connection to the InsightIQ VM and log in using the "administrator" account.
  2. Run the following command to power off the InsightIQ VM:

    sudo shutdown -P now
  3. If possible, create a backup or a snapshot of the InsightIQ VM before proceeding.
  4. In the VM management console, create a new virtual drive to serve as the datastore and connect it to the InsightIQ VM. Refer to your VM server software for instructions on how to perform this step.
  5. Power on the InsightIQ VM.
  6. Open an SSH connection to the InsightIQ VM and log in using the "administrator" account.
  7. Run the following command to open a shell as root:

    sudo -s
  8. Run the following command to list the drives and partitions visible to the InsightIQ VM:

    fdisk –l
  9. Look for the name of the new drive. Make note of the the new drive name, as you will need it to create the new partition, format the drive, and add it. The new drive name will likely be the last entry in the output, such as/dev/sdc.

    You will probably also see a message similar to the following:

    Disk /dev/sdc doesn't contain a valid partition table
    NOTE
    In these steps, we use the name /dev/sdc as an example for the name of the new drive. The name may vary with your configuration but it will always be in /dev.
  10. Run the following command to automatically create a new partition on the new drive, where <drive name> is the name of the new drive:

    echo -e "n\np\n1\n\n\nw" | fdisk <drive name>

    For example:

    echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdc

    Alternatively, you can create the partition manually as follows:
    1. Start an interactive session with fdisk and then create a drive partition by running the following command where <drive name> is the name of the new drive:

      sudo fdisk <drive name>

      For example:

      sudo fdisk /dev/sdc
    2. At the Command prompt, type p to print the partition table.
      IMPORTANT!
      If any defined partitions are displayed in the partition table, do not continue this procedure. Stop here and contact Isilon Technical Support for additional instructions.
    3. Type n to create a new partition.
    4. Type p to create a primary partition.
    5. Type 1 to create partition number one.
    6. When prompted to specify the First and Last cylinders, press Enter to accept the default settings. These steps create one large partition.
    7. Type w to write the partition to the drive and exit the interactive session.
  11. Verify that the partition was created by listing the drives and partitions visible to the InsightIQ VM:

    fdisk -l

    The new datastore should be listed with a valid partition, similar to the following. Note that the new partition will be listed as /dev/sdc1 if the drive name is /dev/sdc.
    Disk /dev/sdc: 107.4 GB, 107374182400 bytes
    255 heads, 63 sectors/track, 13054 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x1c51664b

       Device Boot      Start         End      Blocks   Id   System
    /dev/sdc1               1       13054   104856223+  83   Linux

  12. Make note of the partition name; you will need it later.
  13. Format the new datastore by runing the following command:

    mkfs -t ext3 <device name>

    For example:

    mkfs -t ext3 /dev/sdc1
  14. Create a temporary directory mount location:

    mkdir /datastore_tmp
  15. Stop the InsightIQ application:

    For InsightIQ 2.5 and later:

    iiq_stop

    For InsightIQ 2.1 and earlier:

    sudo stop insightiqNOTE
    Data will not be collected from the monitored clusters while the InsightIQ application is stopped.
  16. For InsightIQ 3.2 and later, please stop the iiq db service:

    sudo service iiq_db stop
  17. Mount the new datastore in a temporary location, where <partition name> is the one you noted in Step 12:

    mount <partition name> /datastore_tmp

    For example:

    mount /dev/sdc1 /datastore_tmp
  18. Verify that the new datastore is mounted and is the correct size as you specified in Step 4.

    df -h

    Output similar to the following appears:
    Filesystem          Size  Used Avail Use% Mounted on
    /dev/sda3           2.3G  1.1G  1.1G  51% /
    tmpfs               939M     0  939M   0% /dev/shm
    /dev/sda1           243M   26M  205M  12% /boot
    /dev/sdb1            63G  545M   60G   1% /datastore
    172.16.132.137:/ifs  40G  3.9G   16G  20% /net/172.16.132.137/ifs
    172.16.132.137:/ifs/.ifsvar/modules/fsa
                         40G  3.9G   16G  20% /net/172.16.132.137/ifs/.ifsvar/modules/fsa
    /dev/sdc1            99G  188M   94G   1% /datastore_tmp
  19. Copy the data from the original datastore to the new datastore:

    cp -rp /datastore/. /datastore_tmp/
  20. Compare the two directories to confirm they are identical:

    diff -rq /datastore /datastore_tmp

    If the directories are identical, you will be returned to a command prompt. Otherwise, a list of differences will be displayed.
    If there are differences, you might need to re-copy the missing files. Contact Isilon Technical Support if you need assistance.
  21. Back up the /etc/fstab entry:

    cp -v /etc/fstab /etc/fstab.bak
  22. Determine the UUID of the new filesystem. This is necessary to complete the next step. Run the below command and note the UUID returned:

    dumpe2fs /dev/sdc1 | grep UUID
  23. Use a text editor to edit the /etc/fstab file to configure InsightIQ to use the new datastore:
    CAUTION!
    The system might be rendered inoperable if configuration files are edited incorrectly. Before editing a configuration file, create a backup of the file so that the file can be restored if necessary.
    1. Open the /etc/fstab file in a text editor.
    2. In the row that contains /datastore as the mount point, prefix the line with a pound "#" to deactivate the old datastore.
    3. Add a new line substituting the UUID of the new datastore determined in step 21.   UUID=<UUID> /datastore ext3 defaults 0 0
      Example /etc/fstab file for InsightIQ 2.5 and later:
      UUID=46f94d79-787a-40b4-b406-6e419f1f5d53 /       ext3    defaults    1 1
      UUID=f2ed68f4-a4d4-4c01-b3c1-cc436b47e3c8 /boo    ext3    defaults    1 2
      UUID=527ffcb4-d9cf-4eb0-b46c-896e8b98907e swap    swap    defaults    0 0
      tmpfs             /dev/shm         tmpfs    defaults        0   0
      devpts            /dev/pts         devpts   gid=5,mode=620  0   0
      sysfs             /sys             sysfs    defaults        0   0
      proc              /proc            proc     defaults        0   0
      # /dev/sdb1         /datastore       ext3     defaults        0   0
      UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /datastore  ext3 defaults   0 0
      Example /etc/fstab file for InsightIQ 2.1 and earlier:# /etc/fstab: static file system information.
      # <file system>   <mount point>   <type>   <options>   <dump> <pass>
      proc              /proc            proc     defaults    0      0
      /dev/sda1         /                ext4     defaults    0      0
      /dev/sda2         swap             swap     defaults    0      0
      # /dev/sdb1         /datastore       ext3     defaults    0      0
      UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /datastore  ext3 defaults   0 0

      Example line before the change:

      /dev/sdb1 /datastore ext3 defaults 0 0

      Example line after the change:

      # /dev/sdb1 /datastore ext3 defaults 0 0
    4. Save the file and exit the text editor.
  24. To implement the new configuration, reboot the InsightIQ VM:

    reboot
  25. Once the VM has rebooted, log back in as "administrator".
  26. Verify the new drive is mounted as /datastore:

    mount | grep datastore

    You should see the new partition mounted

    UUID=<uuid> on /datastore type ext3 (rw)

    For example:

    UUID=79159458-5e18-42d1-96b0-d2c90458bf6b on /datastore type ext3 (rw)
  27. Verify that InsightIQ is working as expected and that historical data from the old datastore is available from the InsightIQ web administration interface.
  28. Run the following command to remove the temporary directory you created in Step 14:

    sudo rmdir /datastore_tmp/
    1. Power off the InsightIQ VM.
    2. On the VM server, remove the old datastore from your InsightIQ VM. The old datastore is normally labeled “Hard disk 2” and is 64 GB in size by default. (It might be different if someone modified the VM.) Refer to your VM server software for instructions on how to perform this step if necessary.

Reverting to the previous datastore

If these changes need to be reverted, you can revert to the old /etc/fstab by running the folliwng command and then rebooting:            
sudo cp -v /etc/fstab.bak /etc/fstab
reboot
Alternatively, you can shut down and revert to a snapshot if one was taken.

Remove the old datatstore

After confirming that the new datastore works as expected and contains all the desired historical data, you can remove the old datastore:

Additional information

Related article:
  • InsightIQ 2.5 and later: Performance best practices guide, 183788
WillSanders
2 Iron

Re: InsightIQ Local /datastore Expansion

Thanks

0 Kudos
WillSanders
2 Iron

Re: InsightIQ Local /datastore Expansion

Can we get this information added to the InsightIQ installation guide?  At the very least can we get some details added about disk space layout for local directories?  Without this KB there is no way to plan how to build the InsightIQ VM structure.

Further, is there a reason that the command "parted" is not in the CentOS release being used for the InsightIQ VM?

Thanks,

Will S.

0 Kudos
jason_f1
1 Copper

Re: InsightIQ Local /datastore Expansion

I just used this KB article to add more storage to an InsightIQ 3.0.1 instance. It worked perfectly. Thanks!

johnsonka
3 Argentium

Re: InsightIQ Local /datastore Expansion

Hello WillSanders,

Thank you for your feedback. I will pass the documentation request along to the team.

As for the "parted" command, the OVA was designed to be very stripped down and easily packaged by engineering, and I cannot speak to their decision to remove certain commands over others; however, you can install any command you would like through the "yum" package manager. I use this functionality a lot myself to add commands to my OVAs to customize them to my needs.

Please let me know if you have any additional questions or concerns.

0 Kudos
tamartin
1 Copper

Re: InsightIQ Local /datastore Expansion

Hi everyone,

The best way to submit documentation feedback and content requests is to send an email to isicontent@emc.com.

WillSanders, I forwarded your request to update the IIQ docs to the documentation feedback team.

Thanks!

Travis

0 Kudos