Avamar: '/' 공간 활용도로 인해 ADME UI가 비어 있고 빈 콘텐츠가 표시됩니다.

로그 파일이 제거되고 용량이 줄어들었으며 결과적으로 UI를 채울 수 있었습니다.
  

root@<avamar-host>:/atoadmin/tmp/atocfg2/#: cp autotapeout2.stat /data01/.
root@<avamar-host>:/atoadmin/tmp/atocfg2/#: rm -rf autotapeout2.stat
root@<avamar-host>:/atoadmin/log/#: df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             7.9G  6.2G  1.4G  83% /
udev                   16G  256K   16G   1% /dev
tmpfs                  16G     0   16G   0% /dev/shm
/dev/sda1             114M   54M   55M  50% /boot
/dev/sda3             1.8T  105G  1.7T   6% /data01
/dev/sda7             1.5G  171M  1.3G  12% /var
/dev/sdb1             1.9T   61G  1.8T   4% /data02
/dev/sdc1             1.9T   60G  1.8T   4% /data03
root@<avamar-host>:/atoadmin/log/#:

다음 작업 후에는 ADMe UI를 재시작하는 것이 좋습니다.

root@<avamar-host>:/data01/#: adme -gui stop

ADMe WEB-UI가 중지되었습니다.

root@<avamar-hostname>:/data01/#: adme -gui start

ADMe WEB-UI 서비스가 성공적으로 시작되었습니다. PID=[122153]
시작하려면 브라우저에서 URL을 가리키십시오. 

https://<avamar-hostname>:8888

root@<avamar-host>:/data01/#:
root@<avamar-host>:/data01/#: adme -gui status

ADMe WEB-UI 서비스 시작 PID=[122153]
Web-UI를 시작하려면 브라우저에서 URL https://< avamar-hostname>:8888로 이동합니다.

root@<avamar-host>:/data01/#:

ADMe UI(https://Avamar-URL:8888)를 열고 로깅하면 ADMe UI에 리소스와 콘텐츠가 없는 빈 페이지가 표시됩니다.

증상:
- ADMe에 빈 흰색 페이지가 표시됩니다.
- 다른 탭이나 보기를 탐색하면 빈 흰색 페이지도 나타나고 내용도 나타나지 않습니다.

조사 후 '/'루트 디렉토리가 100 %로 가득 찬 것으로 나타났습니다.

공간으로 인해 로깅되지 않는 경우 ADME UI를 확인할 수 있습니다.

1. UI 상태
   1. 이는 UI 상태를 확인하는 아래 명령을 사용하여 확인할 수 있습니다.

root@<avamar-hostname>:/atoadmin/#: adme -gui status
/usr/local/avamar/bin/adme[18520]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18528]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18535]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18548]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18554]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18564]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18567]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18568]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18569]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18570]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18571]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18572]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18573]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18574]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18575]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18611]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18612]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18613]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18614]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18615]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18616]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18617]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18618]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18619]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18620]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18621]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18622]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[18623]: echo: write to 1 failed [No space left on device]
/usr/local/avamar/bin/adme[19083]: echo: write to 1 failed [No space left on device]

ADMe WEB-UI service is started PID=[20228]
 To launch Web-UI, point your browser to URL -  https://<avamar-hostname>:8888

root@<avamar-hostname>:/atoadmin/#:

2- df -h
용량을 사용하는 파일을 식별하려면 다음 단계를 수행하십시오.

root@<avamar-hostname>:/atoadmin/tmp/atocfg2/#: df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             7.9G  7.9G     0 100% /
udev                   16G  256K   16G   1% /dev
tmpfs                  16G     0   16G   0% /dev/shm
/dev/sda1             114M   54M   55M  50% /boot
/dev/sda3             1.8T  104G  1.7T   6% /data01
/dev/sda7             1.5G  171M  1.3G  12% /var
/dev/sdb1             1.9T   61G  1.8T   4% /data02
/dev/sdc1             1.9T   60G  1.8T   4% /data03
root@<avamar-hostname>:/atoadmin/tmp/atocfg2/#:

root@<Avamar-hostname>:/atoadmin/#: cd /
root@<Avamar-hostname>://#: perl -e'%h=map{/.\s/;99**(ord$&&7)-$`,$_}`du -hx`;die@h{sort%h}'
0       ./dev
7.7G    .
4.7G    ./usr
2.9G    ./usr/local
2.3G    ./usr/local/avamar
2.1G    ./atoadmin
1.7G    ./atoadmin/tmp
1.6G    ./atoadmin/tmp/atocfg2
1.2G    ./usr/local/avamar/bin
692M    ./usr/local/avamar/lib
508M    ./usr/local/avamar-tomcat-7.0.59
497M    ./usr/local/avamar-tomcat-7.0.59/webapps
437M    ./usr/share
399M    ./usr/local/avamar/tmp
360M    ./usr/java
341M    ./usr/lib64
335M    ./usr/lib
301M    ./opt
294M    ./root
291M    ./root/.avamardata
276M    ./atoadmin/log
201M    ./usr/sbin
188M    ./opt/emc-third-party
182M    ./usr/java/jre1.8.0_131
181M    ./usr/java/jre1.8.0_131/lib
176M    ./usr/java/jre1.8.0_112
175M    ./usr/java/jre1.8.0_112/lib
173M    ./lib
162M    ./lib/modules
160M    ./usr/bin
153M    ./opt/emc-third-party/platform/suse-11-x64/ruby-1.9.1
141M    ./usr/lib64/erlang
136M    ./usr/local/avamar/lib/jetty
127M    ./usr/local/avamar-tomcat-7.0.59/webapps/aam
121M    ./usr/local/avamar-tomcat-7.0.59/webapps/dtlt
115M    ./usr/lib64/erlang/lib
111M    ./usr/share/locale
109M    ./usr/lib/locale

root@<avamar-hostname>:/atoadmin/log/#: ls -lSrh
total 366M
-rwxrwxrwx 1 root root    0 Mar 15 17:06 webui.log.0.lck
-rwxrwxrwx 1 root root    0 May  8 17:27 atoevent.log
-rwxrwxrwx 1 root root    0 Oct 26  2016 1
-rwxrwxrwx 1 root root  13K Apr 14  2017 admbatch-ADM.log
-rwxrwxrwx 1 root root  25K Nov  9 14:47 admbatch-LINUXFS.log
-rwxrwxrwx 1 root root  28K Apr 19  2017 atoevent.log20
-rwxrwxrwx 1 root root  31K Apr 17 03:15 recapture_history.log
-rwxrwxrwx 1 root root  80K Mar 15 17:06 nohup.out
-rwxrwxrwx 1 root root 196K Apr 10 07:46 admbatch-HYPERV2.log
-rwxrwxrwx 1 root root 241K Apr 18  2017 admbatch-HYPERV.log
-rwxrwxrwx 1 root root 346K Apr 27 08:59 atoevent.log3
-rwxrwxrwx 1 root root 637K Apr 13  2016 admbatch-ONDEMAND.log
-rwxrwxrwx 1 root root 671K May  7 15:32 atoevent.log2
-rwxrwxrwx 1 root root 1.2M Apr 27 08:59 admbatch-WINFS.log
-rwxrwxrwx 1 root root 1.3M May  1 16:29 admbatch-SQL1.log
-rwxrwxrwx 1 root root 1.6M May  3 20:08 admbatch-SQL.log
-rwxrwxrwx 1 root root 2.5M Apr  7 02:37 admbatch-HYPERV1.log
-rwxrwxrwx 1 root root 2.7M Apr 20 08:51 admbatch-NDMP.log
-rwxrwxrwx 1 root root 8.9M May  9 11:06 webui.log.0
-rwxrwxrwx 1 root root  11M Jun 13  2017 webui.log.4
-rwxrwxrwx 1 root root  11M Jan 17 19:10 webui.log.1
-rwxrwxrwx 1 root root  11M Aug  8  2017 webui.log.3
-rwxrwxrwx 1 root root  93M May  9 11:15 admbatch-NDMP1.log
-rwxrwxrwx 1 root root 223M Sep 20  2017 webui.log.2


 

이 문제를 추가로 조사한 결과, ADMe가 3일 연속으로 해당 파일에 로깅하여 다음 메시지를 표시한 것으로 확인되었습니다.

root@<avamar-hostname>:/atoadmin/log/#: tail -20 webui.log.2
 >>> Waiting for -migrate phase to begin before initiating the cancel operation 
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
......................................................................................
................................................

이 시나리오에서 ADMe는 Avamar 디스패처 또는 세션에 연결하지 못했고, 유지 보수 명령을 생성할 수 없었으며, 그 후 3일 동안 계속 실패했습니다.

Sep 18, 2017 5:05:34 PM com.avamar.pss.server.ATOServiceImpl runADMECommand
INFO: Output : 0  ERROR!  Exit code 15: Cannot connect to Avamar dispatcher
ERROR: avmaint: cpstatus: cannot connect to server <avamar-hostname> at <avamar-ip>:27000
ERROR: avmaint: gcstatus: cannot        connect to server <avamar-hostname> at <avamar-ip>:27000

 Gathering client/group properties & job stats information...

Sep 18, 2017 5:05:34 PM com.avamar.pss.server.ATOServiceImpl getJobActivities
INFO: Loading Activities for : all from /atoadmin/jobstats/jobactivity.csv
Sep 18, 2017 5:08:37 PM com.avamar.pss.server.ATOServiceImpl runADMECommand
INFO: Output :
 >>> Waiting for -migrate phase to begin before initiating the cancel operation .
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ..
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ...
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ....
 >>> Waiting for -migrate phase to begin before initiating the cancel operation .....
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ......
 >>> Waiting for -migrate phase to begin before initiating the cancel operation .......
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ........
 >>> Waiting for -migrate phase to begin before initiating the cancel operation .........
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ..........
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ...........
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ............
 >>> Waiting for -migrate phase to begin before initiating the cancel operation .............
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ..............
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ...............
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ................
 >>> Waiting for -migrate phase to begin before initiating the cancel operation .................
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ..................
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ...................
 >>> Waiting for -migrate phase to begin before initiating the cancel operation ....................
 >>> Waiting for -migrate phase to begin before initiating the cancel operation .....................

1. 이 문제를 해결하려면 루트 '/' 마운트 지점의 사용량을 100% 미만으로 줄여 ADMe UI가 올바르게 채워지도록 합니다.
2. Dell 지원에 문의하여 파일을 식별하고 식별된 파일을 제거할 수 있는지 확인합니다.  
3. 루트 '/'에서 공백이 제거되고 공백이 100% 미만이면 ADME UI를 다시 시작합니다.

공간 문제를 해결하려면 KB(Knowledge Base)와 유사한 내용을 참조하십시오.

Avamar - Avamar 노드에서 디스크 공간을 많이 차지하는 대용량 파일 또는 디렉토리를 식별하는 방법