UNSOLVED

PetterEdin1

updated

14 years ago

P

PetterEdin1

1 Rookie

•

8 Posts

0

9343

July 24th, 2012 12:00

Avamar single node with capacity full

Hi,

I was called in from vacation today to clear out some old backups on a Avamar 6.0 7.8TB single node that we use as replication target (no clients have it as target).

EMC support was involved with our servicedesk personel earlier today, I'm not shure what they did, but I was told that we needed to delete some backup data that was replicated from the multinode server on the primary site.

When I look at the system now, I can not see that we have a capacity issue on the node anymore;

admin@singlenode:~/>: avmaint nodelist | grep fs-percent

        fs-percent-full="70.7"

        fs-percent-full="68.8"

        fs-percent-full="68.9"

        fs-percent-full="69.0"

        fs-percent-full="68.9"

        fs-percent-full="68.8"

admin@singlenode:~/>: capacity.sh

Top 3 High Change Clients:

--------------------------

Total for all clients                  2381610 mb      100.0%

  client1.domain.com                  2046433 mb       85.9%

  client2.domain.com                      191598 mb        8.0%

  client3.comain.com                      28689 mb        1.2%

The system was left with Maintanance windows scheduler disabled, so I enabled it as a start, gsan is still in degraded mode, will that correct it self after garbage collection have runned successful or am I missing something?

admin@singlenode:~/>: dpnctl status

Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)

dpnctl: INFO: gsan status: degraded

dpnctl: INFO: MCS status: up.

dpnctl: INFO: EMS status: up.

dpnctl: INFO: Backup scheduler status: up.

dpnctl: INFO: dtlt status: up.

dpnctl: INFO: axionfs status: up.

dpnctl: INFO: Maintenance windows scheduler status: enabled.

dpnctl: INFO: Unattended startup status: enabled.

admin@singlenode:~/>: status.dpn

Tue Jul 24 21:09:20 CEST 2012  [SINGLENODE.DOMAIN.COM] Tue Jul 24 19:09:20 2012 UTC (Initialized Wed Sep  7 09:13:38 2011 UTC)

Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk

0.0   10.193.63.100   6.0.1-66  ONLINE fullaccess mhpu+0hpu+0000   2  true   0.14 32560348 10274673  65.7%  66%(onl:4699) 65%(onl:4695) 65%(onl:4696) 65%(onl:4699) 65%(onl:4697) 65%(onl:4695)

Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

 

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0000)

System-Status: ok

Access-Status: admin

 

Last checkpoint: cp.20120724095658 finished Tue Jul 24 12:10:27 2012 after 13m 29s (OK)

Last GC: finished Tue Jul 24 08:15:00 2012 after 00m 00s >> recovered 0.00 KB (MSG_ERR_DISKFULL)

Last hfscheck: finished Tue Jul 24 11:41:07 2012 after 59m 09s >> checked 2052 of 2052 stripes (OK)

 

Maintenance windows scheduler capacity profile is active.

  The backup window is currently running.

  Next backup window start time: Wed Jul 25 20:00:00 2012 CEST

  Next blackout window start time: Wed Jul 25 08:00:00 2012 CEST

  Next maintenance window start time: Wed Jul 25 11:00:00 2012 CEST

Best regards,

Petter

  • ionthegeek

    2 Intern

    •

    2049 Posts

    2312

    0

    Posted July 25th, 2012 05:00

    Whenever an Avamar 6.0 system is in a read-only state, it will report that it is degraded since it can't accept new data.

    This system is currently at the read-only threshold and it looks like the last garbage collection failed because the OS capacity was high. Garbage collection will need to recover enough space to bring the capacity reported in status.dpn down to 62.9% on each partition before the system will return to full access mode and new data can be ingested.

    Could you send me the SR number for this issue in a private message? I would like to review the notes. You can send me a PM from my profile.