VNX: Event Code: 0xfffffffffffffff9 ; Error HC_BE_14505017506 The following domain file(s) does not exist

Control Station failover/failback

1. File Options are disabled in Unisphere

2. Login Warning or Error

   Message :            Logging into 172.28.7.60 encountered an error.
   Full Description :               An error occurred during the login process. The control station did not log in properly.
   Recommended Action :   The login error may have occurred because:
   1. Certificates are not accepted.
   2. Both storage processors or the control station are not accessible.
   3. You have logged in to a File or Block system using a local user account not defined on both the File and Block systems.
   Event Code :       0xfffffffffffffff9

3. nas_checkup error:

   -------------------------------------Errors-------------------------------------
   Storage System : Check for domain and federations health on VNX
   Error HC_BE_14505017506: The following domain file(s) does not exist
             * /nas/http/domain/domain_list
             * /nas/http/domain/domain_master
             * /nas/http/domain/domain_provider_xml
   Action : Contact your service provider and reference EMC Knowledgebase
            article emc273714.

In the most observed cases, Control Station failover or reboot cause the Domain files to get corrupted.

If the control station is failed over to slot_1, then Failback the control station by logging into slot_1 Control Station and rebooting it.

If you have access to Putty session (ssh) to control station (Primary), try the command(Type "yes" when it prompts):

[nasadmin@VNX137 ~]$ /nasmcd/sbin/getreason

If the result shows that:

slot_1 Primary Control station
slot_0 Secondary Control station

You must fail back the control station by rebooting the slot_1 control station. Log in as root (su) and run the following command:

[nasadmin@VNX137 ~]# reboot

This would take a few minutes to reboot the control station and it to come up in the original configuration.

Once the control stations are back on original status, check the domain files:

[nasadmin@VNX137 ~]$ /nasmcd/sbin/getreason
10 - slot_0 primary control station
11 - slot_1 secondary control station
5 - slot_4 contacted
5 - slot_3 contacted

[nasadmin@VNX137 ~]$ ll /nas/http/domain
total 15
drwxr-xr-x 2 apache apache 1024 Jun 27 14:35 bak
drwxr-xr-x 2 apache apache 1024 Nov 14  2013 ldap
-rw-r--r-- 1 apache apache    0 Mar 24 22:04 lock_file
[nasadmin@VNX137 ~]$

Verify the Control Station hostname to be the same as in storage domain:

[nasadmin@VNX137 ~]$ hostname
VNX137
[nasadmin@VNX137 ~]$ hostname -i
10.31.139.137
[nasadmin@VNX137 ~]$ grep initiator /nas/dos/slot_3/boot.cfg
setinitiator cs_hostname=Celerra_VNX137 cs_external_ip=10.31.139.137

[nasadmin@VNX137 ~]$ /nas/sbin/navicli -h spa domain -list
Node:                 VNX_137
IP Address:           10.31.139.141 (Master)
Name:                 SPA
Port:                 80
Secure Port:          443
IP Address:           10.31.139.142
Name:                 SPB
Port:                 80
Secure Port:          443
IP Address:           10.31.139.137
Name:                 Celerra_VNX137
Port:                 80
Secure Port:          443

[nasadmin@VNX137 ~]$

If the external IP of the Primary Control Station is not matching, then follow the Scenario 3 in the article https://support.emc.com/kb/448653 otherwise go further.

As root run below commands:
# /nas/http/webui/bin/domain_admin -i <CS_IP>
# /nas/sbin/naviseccli -h SPA -user sysadmin -password sysadmin -scope 0 domain -messner -add -system <CS_IP>
# /nas/http/webui/bin/update_domain_directory.pl

Then verify the updated domain files:

[nasadmin@VNX137 ~]$ ll /nas/http/domain
total 13
drwxr-xr-x 2 apache apache 1024 Sep  1 10:20 bak
-rw-r--r-- 1 apache apache   64 Sep  8 12:00 domain_list
-rw-r--r-- 1 apache apache  194 Sep  8 12:00 domain_master
-rw-r--r-- 1 apache apache 6764 Sep  8 12:00 domain_provider_xml
-rw------- 1 apache apache   58 Sep  8 12:00 domain_users
-rw-r--r-- 1 apache apache   47 Sep  1 10:20 domain_version
drwxr-xr-x 2 apache apache 1024 Nov 14  2013 ldap
-rw-r--r-- 1 apache apache    0 Aug 23 14:46 lock_file

Now, log in to Unisphere to verify if the alert is cleared and the File options are up.