InsightIQ Local /datastore Expansion

Solution

Perform the following steps to create a new, larger local datastore, configure InsightIQ to use the new datastore, and copy the existing data from the old datastore to the new datastore.

1. Open an SSH connection to the InsightIQ VM and log in using the "administrator" account.
2. Run the following command to power off the InsightIQ VM:
   
   sudo shutdown -P now
   
3. If possible, create a backup or a snapshot of the InsightIQ VM before proceeding.
4. In the VM management console, create a new virtual drive to serve as the datastore and connect it to the InsightIQ VM. Refer to your VM server software for instructions on how to perform this step.
5. Power on the InsightIQ VM.
6. Open an SSH connection to the InsightIQ VM and log in using the "administrator" account.
7. Run the following command to open a shell as root:
   
   sudo -s
   
8. Run the following command to list the drives and partitions visible to the InsightIQ VM:
   
   fdisk –l
   
9. Look for the name of the new drive. Make note of the the new drive name, as you will need it to create the new partition, format the drive, and add it. The new drive name will likely be the last entry in the output, such as/dev/sdc.
   
   You will probably also see a message similar to the following:
   
   Disk /dev/sdc doesn't contain a valid partition table
   NOTE
   In these steps, we use the name /dev/sdc as an example for the name of the new drive. The name may vary with your configuration but it will always be in /dev.
10. Run the following command to automatically create a new partition on the new drive, where is the name of the new drive:
    
    echo -e "n\np\n1\n\n\nw" | fdisk
    
    For example:
    
    echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdc
    
    Alternatively, you can create the partition manually as follows:
    1. Start an interactive session with fdisk and then create a drive partition by running the following command where is the name of the new drive:
       
       sudo fdisk
       
       For example:
       
       sudo fdisk /dev/sdc
       
    2. At the Command prompt, type p to print the partition table.
       IMPORTANT!
       If any defined partitions are displayed in the partition table, do not continue this procedure. Stop here and contact Isilon Technical Support for additional instructions.
    3. Type n to create a new partition.
    4. Type p to create a primary partition.
    5. Type 1 to create partition number one.
    6. When prompted to specify the First and Last cylinders, press Enter to accept the default settings. These steps create one large partition.
    7. Type w to write the partition to the drive and exit the interactive session.
       
11. Verify that the partition was created by listing the drives and partitions visible to the InsightIQ VM:
    
    fdisk -l
    
    The new datastore should be listed with a valid partition, similar to the following. Note that the new partition will be listed as /dev/sdc1 if the drive name is /dev/sdc.
    Disk /dev/sdc: 107.4 GB, 107374182400 bytes
    255 heads, 63 sectors/track, 13054 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x1c51664b
    
       Device Boot      Start         End      Blocks   Id   System
    /dev/sdc1               1       13054   104856223+  83   Linux
    
12. Make note of the partition name; you will need it later.
    
13. Format the new datastore by runing the following command:
    
    mkfs -t ext3
    
    For example:
    
    mkfs -t ext3 /dev/sdc1
    
14. Create a temporary directory mount location:
    
    mkdir /datastore_tmp
    
15. Stop the InsightIQ application:
    
    For InsightIQ 2.5 and later:
    
    iiq_stop
    
    For InsightIQ 2.1 and earlier:
    
    sudo stop insightiqNOTE
    Data will not be collected from the monitored clusters while the InsightIQ application is stopped.
16. For InsightIQ 3.2 and later, please stop the iiq db service:
    
    sudo service iiq_db stop
17. Mount the new datastore in a temporary location, where is the one you noted in Step 12:
    
    mount /datastore_tmp
    
    For example:
    
    mount /dev/sdc1 /datastore_tmp
    
18. Verify that the new datastore is mounted and is the correct size as you specified in Step 4.
    
    df -h
    
    Output similar to the following appears:
    Filesystem          Size  Used Avail Use% Mounted on
    /dev/sda3           2.3G  1.1G  1.1G  51% /
    tmpfs               939M     0  939M   0% /dev/shm
    /dev/sda1           243M   26M  205M  12% /boot
    /dev/sdb1            63G  545M   60G   1% /datastore
    172.16.132.137:/ifs  40G  3.9G   16G  20% /net/172.16.132.137/ifs
    172.16.132.137:/ifs/.ifsvar/modules/fsa
                         40G  3.9G   16G  20% /net/172.16.132.137/ifs/.ifsvar/modules/fsa
    /dev/sdc1            99G  188M   94G   1% /datastore_tmp
19. Copy the data from the original datastore to the new datastore:
    
    cp -rp /datastore/. /datastore_tmp/
    
20. Compare the two directories to confirm they are identical:
    
    diff -rq /datastore /datastore_tmp
    
    If the directories are identical, you will be returned to a command prompt. Otherwise, a list of differences will be displayed.
    If there are differences, you might need to re-copy the missing files. Contact Isilon Technical Support if you need assistance.
    
21. Back up the /etc/fstab entry:
    
    cp -v /etc/fstab /etc/fstab.bak
22. Determine the UUID of the new filesystem. This is necessary to complete the next step. Run the below command and note the UUID returned:
    
    dumpe2fs /dev/sdc1 | grep UUID
23. Use a text editor to edit the /etc/fstab file to configure InsightIQ to use the new datastore:
    CAUTION!
    The system might be rendered inoperable if configuration files are edited incorrectly. Before editing a configuration file, create a backup of the file so that the file can be restored if necessary.
    1. Open the /etc/fstab file in a text editor.
    2. In the row that contains /datastore as the mount point, prefix the line with a pound "#" to deactivate the old datastore.
    3. Add a new line substituting the UUID of the new datastore determined in step 21.   UUID= /datastore ext3 defaults 0 0
       Example /etc/fstab file for InsightIQ 2.5 and later:
       UUID=46f94d79-787a-40b4-b406-6e419f1f5d53 /       ext3    defaults    1 1
       UUID=f2ed68f4-a4d4-4c01-b3c1-cc436b47e3c8 /boo    ext3    defaults    1 2
       UUID=527ffcb4-d9cf-4eb0-b46c-896e8b98907e swap    swap    defaults    0 0
       tmpfs             /dev/shm         tmpfs    defaults        0   0
       devpts            /dev/pts         devpts   gid=5,mode=620  0   0
       sysfs             /sys             sysfs    defaults        0   0
       proc              /proc            proc     defaults        0   0
       # /dev/sdb1         /datastore       ext3     defaults        0   0
       UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /datastore  ext3 defaults   0 0Example /etc/fstab file for InsightIQ 2.1 and earlier:# /etc/fstab: static file system information.
       #            
       proc              /proc            proc     defaults    0      0
       /dev/sda1         /                ext4     defaults    0      0
       /dev/sda2         swap             swap     defaults    0      0
       # /dev/sdb1         /datastore       ext3     defaults    0      0
       UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /datastore  ext3 defaults   0 0
       Example line before the change:
       
       /dev/sdb1 /datastore ext3 defaults 0 0
       
       Example line after the change:
       
       # /dev/sdb1 /datastore ext3 defaults 0 0
       
    4. Save the file and exit the text editor.
       
24. To implement the new configuration, reboot the InsightIQ VM:
    
    reboot
    
25. Once the VM has rebooted, log back in as "administrator".
26. Verify the new drive is mounted as /datastore:
    
    mount | grep datastore
    
    You should see the new partition mounted
    
    UUID= on /datastore type ext3 (rw)
    
    For example:
    
    UUID=79159458-5e18-42d1-96b0-d2c90458bf6b on /datastore type ext3 (rw)
    
27. Verify that InsightIQ is working as expected and that historical data from the old datastore is available from the InsightIQ web administration interface.
28. Run the following command to remove the temporary directory you created in Step 14:
    
    sudo rmdir /datastore_tmp/
    1. Power off the InsightIQ VM.
    2. On the VM server, remove the old datastore from your InsightIQ VM. The old datastore is normally labeled “Hard disk 2” and is 64 GB in size by default. (It might be different if someone modified the VM.) Refer to your VM server software for instructions on how to perform this step if necessary.

Reverting to the previous datastore

sudo cp -v /etc/fstab.bak /etc/fstab

reboot

Remove the old datatstore

Additional information

• InsightIQ 2.5 and later: Performance best practices guide, 183788