Skip to main content
Start a Conversation

Unsolved

lamsarr2

1 Rookie

 • 

10 Posts

9072

July 7th, 2020 07:00

iSilon H400 Error when adding a node in the cluster

I have a iSilon H400 with 4 nodes, 2 Switch Dell S4112–ON. In each node IntA and IntB is connected to the both backend switch (IntA siwtch1 and IntB switch2).

Ext-1 of each node is connected a the backbone switch by 1G.

Ext-2 of each node is connected a the backbone switch by 10G.

Cluster is created succesffully in the Node1.

Now When I join node2 the process goes to the end with status completed
Isilon OneFS/amd64 (XXX-ISILON-2) (ttyu0)
login: root
Password:

But after login by root/password when i try to excute command of on one side i have this message error in both 

"Failed to connect to REST server: [61] Connection refused". And when trying to connect by GUI by admin/password i have "server-side failure with status code 500".

lamsarr2

1 Rookie

 • 

10 Posts

July 7th, 2020 07:00

Node1 Configs:

Please select an option.

Node build: Isilon OneFS v8.1.2.0 B_8_1_2_016(RELEASE)
Node serial number: XXXXXXXXXXXXXXXX

Select an option:
[ 1] Create a new cluster
[ 2] Join an existing cluster
[ 3] Exit wizard and configure manually
[ 4] Reboot into SmartLock Compliance mode
Wizard >>> 1
Do you accept the EULA? [no] yes
Please change the root password from the default.
Please enter new password for root:
Please re-enter password for root:

Password changed.

Please change the UI admin password from the default.
Please enter new password for admin:
Please re-enter password for admin:

Password changed.


Enter a new name for the cluster:
Configure name >>> XXX-ISILON

Cluster name set to XXX-ISILON

Cluster encoding:
[ 1] Windows-SJIS
[ 2] Windows-949
[ 3] Windows-1252
[ 4] EUC-KR
[ 5] EUC-JP
[ 6] EUC-JP-MS
[ 7] UTF-8-MAC
[ 8] UTF-8
[ 9] ISO-8859-1 (Latin-1)
[10] ISO-8859-2 (Latin-2)
[11] ISO-8859-3 (Latin-3)
[12] ISO-8859-4 (Latin-4)
[13] ISO-8859-5 (Cyrillic)
[14] ISO-8859-6 (Arabic)
[15] ISO-8859-7 (Greek)
[16] ISO-8859-8 (Hebrew)
[17] ISO-8859-9 (Latin-5)
[18] ISO-8859-10 (Latin-6)
[19] ISO-8859-13 (Latin-7)
[20] ISO-8859-14 (Latin-8)
[21] ISO-8859-15 (Latin-9)
[22] ISO-8859-16 (Latin-10)
[Enter] Use current encoding: utf-8
Configure encoding >>>
Encoding set to utf-8

Configure interface int-a:
[ 1] Configure netmask
[ 2] Configure int-a IP ranges
[Enter] Keep the current configuration:
Netmask: (not set)
IP ranges: (not set)
Configure interface int-a >>> 1

Enter a new netmask
Configure int-a netmask >>> 255.255.255.0

Configure interface int-a:
[ 1] Configure netmask
[ 2] Configure int-a IP ranges
[Enter] Keep the current configuration:
Netmask: 255.255.255.0
IP ranges: (not set)
Configure interface int-a >>> 2

Configure int-a IP ranges
[ 1] Add an IP range
[ 2] Delete an IP range
[Enter] Keep the current IP ranges:
IP ranges: (not set)
Configure int-a IP ranges >>> 1

Enter the low IP address of the range to add:
Low IP address (add) >>> 10.1.1.10

Enter the high IP address of the range:
High IP address (add) >>> 10.1.1.15

Configure int-a IP ranges
[ 1] Add an IP range
[ 2] Delete an IP range
[Enter] Keep the current IP ranges:
IP ranges: 10.1.1.10-10.1.1.15
Configure int-a IP ranges >>>

Configure interface int-a:
[ 1] Configure netmask
[ 2] Configure int-a IP ranges
[Enter] Keep the current configuration:
Netmask: 255.255.255.0
IP ranges: 10.1.1.10-10.1.1.15
Configure interface int-a >>>

Select the internal interfaces to configure
[ 1] int-a - primary internal interface
[ 2] int-b - secondary internal interface (failover)
[Enter] Exit configuring internal interfaces
Configure internal interfaces >>> 2

Configure interface int-b (failover):
[ 1] Configure netmask
[ 2] Configure int-b IP ranges
[ 3] Configure failover IP ranges
[Enter] Keep the current configuration:
Netmask: (not set)
IP ranges: (not set)
Failover IP ranges: (not set)
Failover: disabled
Configure interface int-b (failover) >>> 1

Type a new netmask
Configure int-b netmask >>> 255.255.255.0

Configure interface int-b (failover):
[ 1] Configure netmask
[ 2] Configure int-b IP ranges
[ 3] Configure failover IP ranges
[Enter] Keep the current configuration:
Netmask: 255.255.255.0
IP ranges: (not set)
Failover IP ranges: (not set)
Failover: disabled
Configure interface int-b (failover) >>> 2

Configure int-b IP ranges
[ 1] Add an IP range
[ 2] Delete an IP range
[Enter] Keep the current IP ranges:
IP ranges: (not set)
Configure int-b IP ranges >>> 1

Enter the low IP address of the range to add:
Low IP address (add) >>> 10.1.2.10

Enter the high IP address of the range:
High IP address (add) >>> 10.1.2.15

Configure int-b IP ranges
[ 1] Add an IP range
[ 2] Delete an IP range
[Enter] Keep the current IP ranges:
IP ranges: 10.1.2.10-10.1.2.15
Configure int-b IP ranges >>>

Configure interface int-b (failover):
[ 1] Configure netmask
[ 2] Configure int-b IP ranges
[ 3] Configure failover IP ranges
[Enter] Keep the current configuration:
Netmask: 255.255.255.0
IP ranges: 10.1.2.10-10.1.2.15
Failover IP ranges: (not set)
Failover: disabled
Configure interface int-b (failover) >>> 3

Configure failover IP ranges
[ 1] Add an IP range
[ 2] Delete an IP range
[Enter] Keep the current IP ranges:
Failover IP ranges: (not set)
Configure failover IP ranges >>> 1

Enter the low IP address of the range to add:
Low IP address (add) >>> 10.1.3.10

Enter the high IP address of the range:
High IP address (add) >>> 10.1.3.15

Configure failover IP ranges
[ 1] Add an IP range
[ 2] Delete an IP range
[Enter] Keep the current IP ranges:
Failover IP ranges: 10.1.3.10-10.1.3.15
Configure failover IP ranges >>>

Configure interface int-b (failover):
[ 1] Configure netmask
[ 2] Configure int-b IP ranges
[ 3] Configure failover IP ranges
[ 4] Disable failover
[Enter] Keep the current configuration:
Netmask: 255.255.255.0
IP ranges: 10.1.2.10-10.1.2.15
Failover IP ranges: 10.1.3.10-10.1.3.15
Failover: enabled
Configure interface int-b (failover) >>>

Select the internal interfaces to configure
[ 1] int-a - primary internal interface
[ 2] int-b - secondary internal interface (failover)
[Enter] Exit configuring internal interfaces
Configure internal interfaces >>>

Configure external subnet
[ 1] mgmt-1 - External interface
[ 2] 10gige-1 - External interface
[Enter] Exit configuring external network.
Configure external subnet >>> 1

Configure interface mgmt-1:
[ 1] Configure netmask
[ 2] Configure MTU
[ 3] Configure mgmt-1 IP ranges
[Enter] Keep the current configuration:
Netmask: (not set)
MTU: 1500
IP ranges: (not set)
Configure interface mgmt-1 >>> 1

Enter a new netmask
Configure mgmt-1 netmask >>> 255.255.255.192

Configure interface mgmt-1:
[ 1] Configure netmask
[ 2] Configure MTU
[ 3] Configure mgmt-1 IP ranges
[Enter] Keep the current configuration:
Netmask: 255.255.255.192
MTU: 1500
IP ranges: (not set)
Configure interface mgmt-1 >>> 3

Configure external IP ranges
[ 1] Add an IP range
[ 2] Delete an IP range
[Enter] Keep the current IP ranges:
IP ranges: (not set)
Configure mgmt-1 IP ranges >>> 1

Enter the low IP address of the range to add:
Low IP address (add) >>> 10.149.195.51

Enter the high IP address of the range:
High IP address (add) >>> 10.149.195.54

Configure external IP ranges
[ 1] Add an IP range
[ 2] Delete an IP range
[Enter] Keep the current IP ranges:
IP ranges: 10.149.195.51-10.149.195.54
Configure mgmt-1 IP ranges >>>

Configure interface mgmt-1:
[ 1] Configure netmask
[ 2] Configure MTU
[ 3] Configure mgmt-1 IP ranges
[Enter] Keep the current configuration:
Netmask: 255.255.255.192
MTU: 1500
IP ranges: 10.149.195.51-10.149.195.54
Configure interface mgmt-1 >>>

Enter default gateway:
Configure default gateway >>> 10.149.195.61

Configure SmartConnect settings
[ 1] SmartConnect zone name
[ 2] SmartConnect service IP
[Enter] Keep the current SmartConnect settings:
SmartConnect zone name: (not set)
SmartConnect service IP: (not set)
Configure SmartConnect settings >>> 1

Enter SmartConnect zone name
SmartConnect zone name >>> XXX-ISILON-SIEGE

Configure SmartConnect settings
[ 1] SmartConnect zone name
[ 2] SmartConnect service IP
[Enter] Keep the current SmartConnect settings:
SmartConnect zone name: XXX-ISILON-SIEGE
SmartConnect service IP: (not set)
Configure SmartConnect settings >>> 2

Enter SmartConnect service IP
SmartConnect service IP >>> 10.149.195.35

Configure SmartConnect settings
[ 1] SmartConnect zone name
[ 2] SmartConnect service IP
[Enter] Keep the current SmartConnect settings:
SmartConnect zone name: XXX-ISILON-SIEGE
SmartConnect service IP: 10.149.195.35
Configure SmartConnect settings >>>

Configure DNS settings
[ 1] DNS servers
[ 2] Search domains
[Enter] Keep current DNS settings:
DNS servers: (not set)
Search domains: (not set)
Configure DNS settings >>> 1

Enter a comma-separated list of DNS servers:
Configure DNS >>> 10.149.3.250,10.149.201.30

Configure DNS settings
[ 1] DNS servers
[ 2] Search domains
[Enter] Keep current DNS settings:
DNS servers: 10.149.3.250, 10.149.201.30
Search domains: (not set)
Configure DNS settings >>> 2

Enter a comma-separated list of search domains:
Search domains >>> XXXlocal.sn

Configure DNS settings
[ 1] DNS servers
[ 2] Search domains
[Enter] Keep current DNS settings:
DNS servers: 10.149.3.250, 10.149.201.30
Search domains: XXXlocal.sn
Configure DNS settings >>>

Configure external subnet
[ 1] mgmt-1 - External interface
[ 2] 10gige-1 - External interface
[Enter] Exit configuring external network.
Configure external subnet >>>

Configure cluster date and time
[ 1] Configure time zone
[ 2] Configure day and time
[Enter] Keep the current date and time: 2020/07/06 13:58:30 UTC.
Configure date >>>

Configure cluster join mode
[ 1] Manual
[ 2] Secure
[Enter] Keep the current join mode: Manual
Configure join mode >>>
Join mode set to Manual.

You have made the following configuration changes:

Cluster name : (not set) -> XXX-ISILON
Encoding : (not set) -> utf-8
int-a : enabled -> enabled
int-a netmask : (not set) -> 255.255.255.0
int-a IP ranges : (not set) -> { 10.1.1.10-10.1.1.15 }
int-b : disabled -> enabled
int-b netmask : (not set) -> 255.255.255.0
int-b IP ranges : (not set) -> { 10.1.2.10-10.1.2.15 }
lpbk : disabled -> enabled
lpbk netmask : (not set) -> 255.255.255.0
lpbk IP ranges : (not set) -> { 10.1.3.10-10.1.3.15 }
mgmt-1 netmask : (not set) -> 255.255.255.192
mgmt-1 IP range : (not set) -> { 10.149.195.51-10.149.195.54 }
mgmt-1 gateway : (not set) -> 10.149.195.61
SmartConnect zone name : (not set) -> XXX-ISILON-SIEGE
SmartConnect service IP: (not set) -> 10.149.195.35
DNS servers : (not set) -> { 10.149.3.250, 10.149.201.30 }
Search domains : (not set) -> { XXXlocal.sn }

Do you wish to commit these changes? [yes]
Commit changes? >>> yes
arp: writing to routing socket: Device busy
arp: writing to routing socket: Device busy
No matching processes were found
Stopping syslogd.
Waiting for PIDS: 1915.
Starting syslogd.
Executing script isi_led
Executing script isi_post_network
Pre-IFS daemons: isi_watchdog_d isi_hangdump
Ethernet Back-End. isi_eth_mixer_d service DISABLED
isi_flexnet_d.
Executing script mountcritremote
Executing script isi_mcp
Starting isi_mcp.
Executing script virecover
Executing script pwcheck
Executing script tmp
Executing script dmesg
Executing script accounting
Executing script devfs
Executing script archdep
Executing script motd
Executing script hostapd
Executing script gptboot
Executing script nfsclient
Executing script isi_mount_ifs
Updating motd:Mounting IFS filesystem...
.
Executing script cleartmp
Executing script abi
Executing script statd
Clearing /tmp (X related).
Executing script lockd
Executing script localpkg
Executing script SERVERS
Executing script DAEMON
Executing script apm
Executing script ntpd
Executing script lpd
Executing script rarpd
Executing script sdpd
Executing script powerd
Executing script nscd
Executing script moused
Executing script mountlate
Executing script utx
Executing script local
Executing script rtadvd
Mounting late file systems:.
Executing script bootparams
Executing script apmd
Executing script swaplate
Executing script LOGIN
Executing script cron
Executing script othermta
Executing script sendmail
Executing script inetd
Starting cron.
Executing script msgs
Executing script jail
Executing script syscons
Executing script bgfsck
Executing script securelevel
-->/dev/da16s1e
-->/dev/da15s1e
-->/dev/da14s1e
-->/dev/da13s1e
-->/dev/da12s1e
-->/dev/da11s1e
-->/dev/da10s1e
-->/dev/da9s1e
-->/dev/da8s1e
-->/dev/da7s1e
-->/dev/da6s1e
-->/dev/da5s1e
-->/dev/da4s1e
-->/dev/da3s1e
-->/dev/da2s1e

Formatting drives...

N/A N/A Bay A0/da9 100% Bay A1/da2 100%
Bay A2/da10 100% Bay B0/da3 100% Bay B1/da11 100% Bay B2/da4 100%
Bay C0/da12 100% Bay C1/da5 100% Bay C2/da13 100% Bay D0/da6 100%
Bay D1/da14 100% Bay D2/da7 100% Bay E0/da15 100% Bay E1/da8 100%
Bay E2/da16 100%
Executing script isi_post_ifs
a 244b5494.0
a 2c543cd1.0
a 38ae8eda.0
a 415660c1.0
a 653b494a.0
a 6a909d98.0
a BaltimoreCyberTrustRoot.pem
a DigiCertHighAssuranceEV-RootCA.pem
a DigiCertSHA2HighAssuranceServerCA.pem
a GeoTrust_Global_CA.pemx 244b5494.0
a GoogleInternetAuthority_G3.pem
a VerisignClass3PublicPrimaryCertificationAuthority.pem

x 2c543cd1.0
x 38ae8eda.0
x 415660c1.0
x 653b494a.0
x 6a909d98.0
x BaltimoreCyberTrustRoot.pem
x DigiCertHighAssuranceEV-RootCA.pem
x DigiCertSHA2HighAssuranceServerCA.pem
x GeoTrust_Global_CA.pem
x GoogleInternetAuthority_G3.pem
x VerisignClass3PublicPrimaryCertificationAuthority.pem
kern.cam.clear_wce_on_resets: 0 -> 1
kern.cam.lost_cache_downs_drive: 0 -> 1
Upgrading Source Records...
Upgrading Target Records...
Upgrading policies...
Upgrading siq global config...
+ BASEDIR=/ifs/.ifsvar/upgrade
+ BASESCRIPT=/ifs/.ifsvar/upgrade/isi_database_delete.sh
+ LOCKFILE=/ifs/.ifsvar/upgrade/isi_database_delete.sh.lock
+ FLAGFILE=/ifs/.ifsvar/upgrade/isi_database_delete.sh.flag
+ ULOG=/usr/sbin/isi_upgrade_ulog
+ DBFILE=/ifs/.ifsvar/modules/stats/persistence.db
+ [ -e /ifs/.ifsvar/upgrade/isi_database_delete.sh.flag ]
+ [ 0 -eq 0 ]
+ exec lockf -k /ifs/.ifsvar/upgrade/isi_database_delete.sh.lock /usr/share/upgrade/event-actions/post-commit/isi_database_delete.sh under-lock
+ BASEDIR=/ifs/.ifsvar/upgrade
+ BASESCRIPT=/ifs/.ifsvar/upgrade/isi_database_delete.sh
+ LOCKFILE=/ifs/.ifsvar/upgrade/isi_database_delete.sh.lock
+ FLAGFILE=/ifs/.ifsvar/upgrade/isi_database_delete.sh.flag
+ ULOG=/usr/sbin/isi_upgrade_ulog
+ DBFILE=/ifs/.ifsvar/modules/stats/persistence.db
+ [ -e /ifs/.ifsvar/upgrade/isi_database_delete.sh.flag ]
+ [ 1 -eq 0 ]
+ /bin/rm -f /ifs/.ifsvar/modules/stats/persistence.db
+ /usr/sbin/isi_upgrade_ulog -l 42 'Deleted DB file'
+ /usr/bin/isi_gconfig -t stats-config -R history
+ /usr/sbin/isi_upgrade_ulog -l 48 'Reset stats_config history'
+ /usr/bin/touch /ifs/.ifsvar/upgrade/isi_database_delete.sh.flag
+ /usr/sbin/isi_upgrade_ulog -l 52 'Touched flagfile'
+ /usr/bin/killall isi_stats_hist_d
+ /usr/bin/killall isi_stats_d
+ /usr/sbin/isi_upgrade_ulog -l 58 'Sent TERM to isi_stats_hist_d and isi_stats_d'
Please run 'isi upgrade patches --help' for patch system usage
Please run 'isi upgrade patches --help' for patch system usage
Please run 'isi upgrade patches --help' for patch system usage
Please run 'isi upgrade patches --help' for patch system usage
Cleared upgrade config
This platform is not supported by this script
request: STATUS, task:STATUS_SYNC, hash: (null)
Executing script isi_firmware_versions
Save firmware status
completed

Isilon OneFS/amd64 (XXX-ISILON-1) (ttyu0)

login:

-----

Node2 Configs:


Please select an option.

Node build: Isilon OneFS v8.1.2.0 B_8_1_2_016(RELEASE)
Node serial number: XXXXXXXXXXX

Select an option:
[ 1] Create a new cluster
[ 2] Join an existing cluster
[ 3] Exit wizard and configure manually
[ 4] Reboot into SmartLock Compliance mode
Wizard >>> 2

Select the cluster you want to join
Index Name Version Status
----- ---------- ------------ ------------------------------------------------
1 XXX-ISILON B_8_1_2_016R available
[Enter] Refresh the list
Join cluster >>> 1
Joining cluster: 'XXX-ISILON'
Joining status: Beginning join to cluster XXX-ISILON
Joining status: Beginning join to cluster XXX-ISILON
Joining status: Checking version compatibility with cluster XXX-ISILON
Joining status: Versions compatible
Joining status: Joined cluster XXX-ISILON as devid 2
arp: writing to routing socket: Device busy
arp: writing to routing socket: Device busy
No matching processes were found
Stopping syslogd.
Waiting for PIDS: 1878.
Starting syslogd.
Executing script isi_led
Executing script isi_post_network
Pre-IFS daemons: isi_watchdog_d isi_hangdump
Ethernet Back-End. isi_eth_mixer_d service DISABLED
isi_flexnet_d.
Executing script mountcritremote
Executing script isi_mcp
Starting isi_mcp.
Executing script virecover
Executing script pwcheck
Executing script tmp
Executing script dmesg
Executing script accounting
Executing script devfs
Executing script archdep
Executing script motd
Executing script hostapd
Executing script gptboot
Executing script nfsclient
Executing script isi_mount_ifs
Mounting IFS filesystem...
Updating motd:.
Executing script cleartmp
Executing script abi
Executing script statd
Clearing /tmp (X related).
Executing script lockd
Executing script localpkg
Executing script SERVERS
Executing script DAEMON
Executing script apm
Executing script ntpd
Executing script lpd
Executing script rarpd
Executing script sdpd
Executing script powerd
Executing script nscd
Executing script moused
Executing script mountlate
Executing script utx
Executing script local
Executing script rtadvd
Mounting late file systems: Executing script bootparams
.
Executing script apmd
Executing script swaplate
Executing script LOGIN
Executing script cron
Executing script othermta
Executing script sendmail
Executing script inetd
Starting cron.
Executing script msgs
Executing script jail
Executing script syscons
Executing script bgfsck
Executing script securelevel
-->/dev/da16s1e
-->/dev/da15s1e
-->/dev/da14s1e
-->/dev/da13s1e
-->/dev/da12s1e
-->/dev/da11s1e
-->/dev/da10s1e
-->/dev/da9s1e
-->/dev/da8s1e
-->/dev/da7s1e
-->/dev/da6s1e
-->/dev/da5s1e
-->/dev/da4s1e
-->/dev/da3s1e
-->/dev/da2s1e

Formatting drives...

N/A N/A Bay A0/da9 100% Bay A1/da2 100%
Bay A2/da10 100% Bay B0/da3 100% Bay B1/da11 100% Bay B2/da4 100%
Bay C0/da12 100% Bay C1/da5 100% Bay C2/da13 100% Bay D0/da6 100%
Bay D1/da14 100% Bay D2/da7 100% Bay E0/da15 100% Bay E1/da8 100%
Bay E2/da16 100%
mnv0: HW link up event
mnv0: transport link up
mnv0: peer up
Executing script isi_post_ifs
request: STATUS, task:STATUS_SYNC, hash: (null)
Executing script isi_firmware_versions
Save firmware status
completed

Isilon OneFS/amd64 (XXX-ISILON-2) (ttyu0)

login:

Phil.Lam

3 Apprentice

 • 

622 Posts

July 7th, 2020 14:00

@lamsarr2,

Have you resolved the REST API issue?  If not, try rebooting node 2?

DELL-Josh Cr

Moderator

 • 

9.4K Posts

July 7th, 2020 14:00

1. check if the isi_papi_d service is enabled : 

# isi services -a isi_papi_d


2. if the isi_papi_d service is disabled, enable it by running the following command : 

# isi services -a isi_papi_d
Service 'isi_papi_d' is enabled.


3. if the isi_papi_d service was already enabled, check the /var/log/apache2/webui_httpd_error.log for errors such as this : 

2018-09-24T11:47:56Z <18.3> PDC-NAS-ISN01-20 httpd[80131]: [:error] [pid 80131:tid 34676253696] (20)Not a directory: [client 192.168.10.10:37994] FastCGI: failed to connect to server "/usr/sbin/isi_papi_d": socket file descriptor (1712) is larger than FD_SETSIZE (1024), you probably need to rebuild Apache with a larger FD_SETSIZE
2018-09-24T11:47:56Z <18.3> PDC-NAS-ISN01-20 httpd[80131]: [:error] [pid 80131:tid 34676253696] [client 10.0.2.63:37994] FastCGI: incomplete headers (0 bytes) received from server "/usr/sbin/isi_papi_d"
2018-09-24T11:47:56Z <18.3> PDC-NAS-ISN01-20 httpd[80131]: [:error] [pid 80131:tid 34676253696] [client 10.0.2.63:37994] FastCGI: do_work() failed with ret 500 for request GET /platform/1/cluster/config HTTP/1.1


if you see the above error, you should restart the isi_papi_d service by running the below commands : 

# isi services -a isi_papi_d disable

# isi services -a isi_papi_d enable

 

lamsarr2

1 Rookie

 • 

10 Posts

July 8th, 2020 06:00

Hi PhilLam

I also restarted Node2, but no change. 

lamsarr2

1 Rookie

 • 

10 Posts

July 8th, 2020 06:00

Hi Josh,

I executed the commands and checked the service Service 'isi_papi_d' is enabled.
And in "/var/log/apache2/webui_httpd_error.log" we have below the Output of the executed command. Then we have disabled and reactivated the "isi_papi_d" service and the problem still remains. We still have errors in the console ("UVS-ISILON-1# isi status
Failed to connect to REST server: [61] Connection refused")
and in the WebUI "erver-side failure with status code 500".

I also restarted Node 2, but no change.

Jul 6 13:15:34 newsyslog[866]: logfile first created
2020-07-06T14:07:32Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678856704] (61)Connection refused: [client 10.149.196.118:51291] FastCGI: failed to connect to server "/usr/sbin/isi_papi_d": connect() failed, referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:07:32Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678856704] [client 10.149.196.118:51291] FastCGI: incomplete headers (0 bytes) received from server "/usr/sbin/isi_papi_d", referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:07:32Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678856704] [client 10.149.196.118:51291] FastCGI: do_work() failed with ret 500 for request GET /platform/3/event/eventgroup-occurrences?_dc=1594044434091&ignore=false&resolved=false&sort=severity&dir=ASC&limit=1 HTTP/1.1, referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:07:48Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678864896] (61)Connection refused: [client 10.149.196.118:51299] FastCGI: failed to connect to server "/usr/sbin/isi_papi_d": connect() failed, referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:07:48Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678864896] [client 10.149.196.118:51299] FastCGI: incomplete headers (0 bytes) received from server "/usr/sbin/isi_papi_d", referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:07:48Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678864896] [client 10.149.196.118:51299] FastCGI: do_work() failed with ret 500 for request GET /platform/3/event/eventgroup-occurrences?_dc=1594044450093&ignore=false&resolved=false&sort=severity&dir=ASC&limit=1 HTTP/1.1, referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:08:05Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678867968] (61)Connection refused: [client 10.149.196.118:51279] FastCGI: failed to connect to server "/usr/sbin/isi_papi_d": connect() failed, referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:08:05Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678867968] [client 10.149.196.118:51279] FastCGI: incomplete headers (0 bytes) received from server "/usr/sbin/isi_papi_d", referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:08:05Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678867968] [client 10.149.196.118:51279] FastCGI: do_work() failed with ret 500 for request GET /platform/3/event/eventgroup-occurrences?_dc=1594044467089&ignore=false&resolved=false&sort=severity&dir=ASC&limit=1 HTTP/1.1, referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:08:22Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678858752] (61)Connection refused: [client 10.149.196.118:51311] FastCGI: failed to connect to server "/usr/sbin/isi_papi_d": connect() failed, referer: https://10.149.195.51:8080/Status?loadState=1
2020-07-06T14:08:22Z <18.3> XXX-ISILON-1 httpd[5406]: [:error] [pid 5406:tid 34678858752] [client 10.149.196.118:51311] FastCGI: incomplete headers (0 bytes) received from server "/usr/sbin/isi_papi_d", referer: https://10.149.195.51:8080/Status?loadState=1

Phil.Lam

3 Apprentice

 • 

622 Posts

July 12th, 2020 23:00

@lamsarr2 ,

Have you engaged support?

lamsarr2

1 Rookie

 • 

10 Posts

July 13th, 2020 01:00

Yes, I opened a SR and we worked with dell/EMC support for two weeks on this issue but they couldn't solve the problem.

As far as troubleshooting the Dell/EMC support has done all the possible troubleshooting that he could and he had reached the end of it. After all nodes was re-formatted multiple time and tried with different of OneFS 8.1.2.0 and 8.2.0.0, configured with or without DNS and confirmed front and backend connectivity but finaly the new deployment issue could not be resolved.

Phil.Lam

3 Apprentice

 • 

622 Posts

July 13th, 2020 08:00

@lamsarr2,

Can you try commands with "-t 300" option? it means "timeout 300 seconds".  The default is 60 seconds I think.

 

Example:

isi -t 300 status 

khayamanjam

4 Posts

March 8th, 2021 17:00

were you able to solve the issue? I am seeing the same problem.

Phil.Lam

3 Apprentice

 • 

622 Posts

March 12th, 2021 12:00

@khayamanjam 

{ISILON} 8.0+ Cannot use the OneFS Web Interface due to " server-side failure with status code 500" error
https://www.dell.com/support/kbdoc/000167823/cannot-use-the-onefs-webui-server-side-failure-with-status-code-500?lang=en 

lamsarr2

1 Rookie

 • 

10 Posts

March 15th, 2021 01:00

Hi,

I was resolved my issue by changing the backend swicth role at leaf.

smartfabric l3fabric enable role LEAF

davemonstermt

1 Message

March 18th, 2021 08:00

Hello!

Thanks for posting this resolution, this just saved a greenfield install that my colleague and I have been bashing our faces into for 2 days straight with multiple node re-image attempts.

 

All in, we had to perform the following config on both of the Z9100 QSFP Switches used for internal connectivity on this H400 cluster with QSFP breakout cables:

# config

# feature auto-breakout

# write memory

# smartfabric l3fabric enable role LEAF

 

Thanks for sharing!

- Dave

jdrabick

1 Rookie

 • 

4 Posts

April 1st, 2021 12:00

Just wanted to throw my THANKS!! in here as well.

Been a while since I've worked on a physical Isilon install and in the past if the Back-End switches are only being used for internal/back-end traffic they were always "zero-touch".  Well I guess that is not the case anymore.

4 x Node A200 install with 2 x Dell S4112-ON for back-end/internal traffic ONLY.

Created cluster with first node and everything seemed normal.  As soon as I added the 2nd node into the cluster everything broke.  WebUI error, no auth service and REST error when logging into CLI...ect.  I reformatted and tried again, but same thing.

Called support (ultimately not much help initially) and came across this post while waiting on triage to get me to a support engineer.

As soon as I enabled SmartFabric on the switched, everything worked as I would expect it to.

I checked the latest solve documentation, and it does mention enabling SmartFabric, but is a little confusing, because it makes it sound like you only need to enable SFS if using the switches in a Leaf-Spine topology.  Not the case.  It needs to be enabled regardless!!

Was this post helpful?

View All

No Events found!

Top