UNSOLVED

johhen2

updated

17 years ago

J

johhen2

4 Posts

0

1243

June 15th, 2009 06:00

Deduplication for Exchange and SQL

How much compression/deduplication can we expect on Exchange and SQL databases?

We see increasing (new bytes) by up to 6-7% in one day, but we dont have that much new data coming into our mail system.

Anyway to tune settings for to decrease the amount of new bytes everyday? Server utilization is now 89,7%

Brgds,
Henrik Johansen
  • johhen2

    4 Posts

    672

    0

    Posted June 15th, 2009 08:00

    I get following message when trying to view solution:

    Unable to Display Solution

    The solution you requested either could not be found or is not available for viewing
  • 672

    0

    Posted June 15th, 2009 08:00

    That could be because I've just updated it and it's awaiting review.

    Here's a copy/paste:-


    Symptoms

    The Avamar system is full or approaching full capacity

    Resolution
    Aside from preventing any new backups, there are two options available.
    1) Expand the system by adding new nodes to it please contact your EMC sales representative or EMC partner for details).

    2) Reduce the amount of data stored on the system by tuning retention policies.

    In order to identify the clients which are contributing the most amount of data change on the Avamar system we use the script 'capacity.sh'.

    The script can help us analyze the amount of data added and removed from the system on a daily basis and to understand which clients cause the greatest amount of data change on the system in order to tune retention periods.

    To run the script:-
    ===================
    a) Login to the utility node as the admin user
    b) run /usr/local/avamar/bin/capacity.sh

    NOTE: In older versions of Avamar, this script is not present on the systme and needs to be manually copied to it. See below for details.

    If the script is not present in /usr/local/avamar/bin then download it using the following link
    ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/capacity.sh



    To install and execute the script:

    1) Upload capacity.sh to the Avamar utility node using SCP on a Linux command line or with WinSCP on Windows.

    2) Place the script in the /home/admin directory.

    3) Log in to the Avamar system as the 'admin' user.

    4) Change to the directory where you uploaded the script.

    5) Copy the script to the /home/admin directory: cp sched.sh /home/admin/
    4) Change the permissions to allow the script to be executed:

    chmod 744 /home/admin/capacity.sh
    5) Run the script: ./capacity.sh

    Understanding the output from the script:

    The top part of the script output shows the date, the amount of new data added, the number of backups taken, the amount of data removed by garbage collection and the net change rate on the system (data added minus data removed).
    This section gives you an idea of what is happening on a day by day basis. You can use this to understand, at a glance, on which days experienced large increases in backup data and how data has been removed in the garbage collection sessions.

    It will be very clear if there were an unusual amount (or lack of) backup data or if garbage collection failed to run on a particular day.

    Date New Data #BU Removed #GC Net Change
    ---------- ---------- ----- ---------- ----- ----------
    2009-03-04 383236 mb 464 -33167 mb 1 350069 mb
    2009-03-05 505260 mb 649 -56577 mb 1 448682 mb
    2009-03-06 82157 mb 228 -197873 mb 2 -115715 mb
    2009-03-07 499756 mb 434 0 mb 499756 mb
    2009-03-08 243758 mb 328 -98435 mb 1 145322 mb
    2009-03-09 126805 mb 437 -86354 mb 1 40451 mb
    2009-03-10 233511 mb 435 -86748 mb 1 146762 mb
    2009-03-11 152480 mb 140 -83876 mb 1 68603 mb
    ---------- ---------- ----- ---------- ----- ----------
    Average 197349 mb -62578 mb 134770 mb


    The bottom part of the script shows which clients added the most data in the last day.
    Below we see that 7923mb was added to the system in the last day. We are given a breakdown of this amount by the three clients which most contributed to this new data.

    In this case, if we needed to reduce retention periods to lower the amount of data stored on the system we would concentrate on these clients.

    Top Three High Change Clients:
    --------------------------
    Total for all clients 7923 mb 100.0%
    aaaaa.xxxx.emc.com 2381 mb 30.1%
    bbbbb.xxxx.emc.com 1990 mb 25.1%
    ccccc.xxxx.emc.com 1282 mb 16.2%



    Further information:
    1) View the output with courier font for best results.
    2) Run ./capacity.sh --help for further options.
    3) mb as noted in the script should actually be taken to be'Mb'
  • 672

    0

    Posted June 15th, 2009 08:00

    Is the mail system the only client backing up to Avamar?

    Take a closer look at where most of the data is coming from by running the capacity.sh script

    http://solutions.emc.com/emcsolutionview.asp?id=esg103967
  • johhen2

    4 Posts

    672

    0

    Posted June 15th, 2009 08:00

    To answer your first question, no its not the only client, but we have only one client running backup of exchange, and this is the server getting most new bytes.

    Backup of windows filserver have lesse then 0,05% new bytes each day.


    How do I run Capacity.sh script?

    Brgds,
    Henrik
  • johhen2

    4 Posts

    672

    0

    Posted June 15th, 2009 09:00

    Date New Data #BU Removed #GC Net Change
    ---------- ---------- ----- ---------- ----- ----------
    2009-05-16 4810 mb 10 -2801 mb 1 2009 mb
    2009-05-17 2526 mb 10 -1702 mb 1 824 mb
    2009-05-18 9364 mb 10 -406 mb 1 8958 mb
    2009-05-19 8081 mb 10 -471 mb 1 7609 mb
    2009-05-20 5715 mb 10 -391 mb 1 5324 mb
    2009-05-21 4156 mb 10 -382 mb 1 3774 mb
    2009-05-22 9541 mb 10 -472 mb 1 9068 mb
    2009-05-23 2504 mb 10 -653 mb 1 1851 mb
    2009-05-24 3642 mb 10 -238 mb 1 3403 mb
    2009-05-25 13414 mb 10 -780 mb 1 12633 mb
    2009-05-26 5779 mb 10 -7903 mb 1 -2123 mb
    2009-05-27 8601 mb 10 -20300 mb 1 -11698 mb
    2009-05-28 7398 mb 10 -6304 mb 1 1093 mb
    2009-05-29 6728 mb 10 -5366 mb 1 1362 mb
    2009-05-30 3019 mb 10 -6624 mb 1 -3604 mb
    2009-05-31 2144 mb 10 -7402 mb 1 -5258 mb
    2009-06-01 4954 mb 10 -7243 mb 1 -2288 mb
    2009-06-02 15413 mb 10 -9346 mb 1 6067 mb
    2009-06-03 27504 mb 10 -2937 mb 1 24567 mb
    2009-06-04 8731 mb 10 -1953 mb 1 6777 mb
    2009-06-05 7384 mb 10 -3248 mb 1 4136 mb
    2009-06-06 3110 mb 10 -4570 mb 1 -1459 mb
    2009-06-07 4432 mb 10 -2497 mb 1 1935 mb
    2009-06-08 6608 mb 10 -2219 mb 1 4388 mb
    2009-06-09 11119 mb 10 -2763 mb 1 8356 mb
    2009-06-10 7419 mb 10 -2222 mb 1 5196 mb
    2009-06-11 7631 mb 10 -4059 mb 2 3572 mb
    2009-06-12 9352 mb 10 -2206 mb 1 7145 mb
    2009-06-13 5029 mb 10 -3791 mb 1 1237 mb
    2009-06-14 3114 mb 10 -7612 mb 1 -4497 mb
    2009-06-15 2002 mb 3 -9302 mb 1 -7300 mb
    ---------- ---------- ----- ---------- ----- ----------
    Average 7136 mb -4134 mb 3002 mb

    Top 3 High Change Clients:
    --------------------------
    awk: cmd. line:2: warning: escape sequence `\%' treated as plain `%'
    Total for all clients 221241 mb 100.0%
    mail1.domain.com 187167 mb 84.6%
    oradb2.domain.com 12597 mb 5.7%
    fileserver.domain.com 12543 mb 5.7%
  • ble1

    6 Operator

    •

    14354 Posts

    •

    56186 Points

    672

    0

    Posted June 15th, 2009 13:00

    Your server is pretty much utilized and might indicate sizing issue in first place. As to growth, this might be an issue with size of hash cache as well. If your hash cache is not big enough (modified) to get all chunks then you will always see some more growth there. But yes, it seems as it is growing for you and current solution might not be in place for you. Is that AVE that you use? You may wish to perform (or support for you) some aggressive garbage collection you to see if you can free up some space. Also, newer version will give you over web prediction on when your server might get into red zone (even now at 90% I would see that as red zone). If this is AVE, do you know if read-only value has been decreased since initial install?
  • 672

    0

    Posted June 16th, 2009 01:00

    Ok, we can see from this that over the last two days the net data change on the server has been negative. If this trend continues the Avamar storage utilisation should steadily reduce.

    If the "Net Change" becomes positive again then you'll need to revisit the retention policies and reduce them some more (or expand the system) .

    No need to worry about the client hash cache. It doesn't have any effect on the amount of data sent to the server. It simply reduces the amount of time the client takes to scan the dataset.
  • ble1

    6 Operator

    •

    14354 Posts

    •

    56186 Points

    672

    0

    Posted June 16th, 2009 02:00

    No need to worry about the client hash cache. It
    doesn't have any effect on the amount of data sent to
    the server. It simply reduces the amount of time the
    client takes to scan the dataset.

    You are right... I mixed up this with something else when posting before... mea culpa.