PowerFlex 4.X Firmware Update Fails With Invalid Byte Sequence In US-ASCII

Scenario

The cluster was undergoing a Firmware upgrade performed using the Resource Group.

Symptoms

The deployment.log from the Job-ID reports the firmware update failed, and the playbook is running. Note that the command in the playbook is set to run "/usr/sbin/sysctl -p":

ERROR [2024-01-25T20:21:33.358478] 25824: service_deployment.rb:6084:in `process_firmware_update': Encountered an error during firmware update: ArgumentError: invalid byte sequence in US-ASCII
["org/jruby/RubyString.java:4727:in `scan'", "/opt/asm-deployer/lib/asm/provider/elementmanager/scaleio.rb:9762:in `update_sysctl_parameter'", "/opt/asm-deployer/lib/asm/service_deployment.rb:5909:in `block in process_firmware_update'", "org/jruby/RubyArray.java:1865:in `each'", "/opt/asm-deployer/lib/asm/service_deployment.rb:5731:in `process_firmware_update'", "/opt/asm-deployer/lib/asm/service_deployment.rb:504:in `process'", "/opt/asm-deployer/lib/asm.rb:369:in `block in process_deployment'", "org/jruby/RubyArray.java:1907:in `each_slice'", "/opt/asm-deployer/lib/asm.rb:351:in `block in process_deployment'", "/opt/dependencies/dell-asm-util/lib/asm/util.rb:22:in `block in execute_async'"]
...
INFO  [2024-01-25T20:22:45.017056] 25824: service_deployment.rb:1795:in `process_ansible_playbook': Starting processing playbook: [{"hosts"=>"all", "become"=>true, "vars"=>{"overcommit_ratio"=>100, "overcommit_memory"=>2}, "tasks"=>[{"name"=>"Replace unvalid values in sysctl conf", "replace"=>{"path"=>"/etc/sysctl.conf", "regexp"=>"^net.ipv4.conf.(ens192.log_martians.*)$", "replace"=>"#net.ipv4.conf.\\1 # PFXM REMOVED INVALID OPTION"}}, {"name"=>"Replace unvalid values in sysctl conf", "replace"=>{"path"=>"/etc/sysctl.conf", "regexp"=>"^net.ipv4.conf.(ens32.log_martians.*)$", "replace"=>"#net.ipv4.conf.\\1 # PFXM REMOVED INVALID OPTION"}}, {"name"=>"update threshold1", "ansible.posix.sysctl"=>{"name"=>"net.ipv4.neigh.default.gc_thresh1", "value"=>8192}}, {"name"=>"update threshold2", "ansible.posix.sysctl"=>{"name"=>"net.ipv4.neigh.default.gc_thresh2", "value"=>16384}}, {"name"=>"update threshold3", "ansible.posix.sysctl"=>{"name"=>"net.ipv4.neigh.default.gc_thresh3", "value"=>32768}}, {"name"=>"Update overcommit ratio for SDS", "ansible.posix.sysctl"=>{"name"=>"vm.overcommit_ratio", "value"=>"{{ overcommit_ratio }}"}}, {"name"=>"Update overcommit_memory ratio for SDS", "ansible.posix.sysctl"=>{"name"=>"vm.overcommit_memory", "value"=>"{{ overcommit_memory }}"}}, {"name"=>"Apply sysctl changes", "command"=>{"argv"=>["
]}}]}] with inventory:

... When running the command directly on the node, the following error may be seen:

[root@node3 etc]# /usr/sbin/sysctl -p
...
sysctl: /etc/sysctl.conf(91): invalid syntax, continuing...

Impact

The Firmware update is not able to proceed on the affected node.

Root Cause

The issue stems from a non-ASCII or special character in the sysctl.conf file, which is not compatible with the expected US-ASCII character encoding used during the sysctl processing. When the system attempts to read and apply the configurations from sysctl.conf, it encounters this invalid character sequence, leading to an "invalid byte sequence in US-ASCII" error. The specific reason for this character's inclusion remains unclear. 

Workaround

1- Back up the original sysctl.conf file on the affected node:

 cp /etc/sysctl.conf /etc/sysctl.conf.backup

2- Copy a sysctl.conf file from a node where it is known to be functioning correctly:

 scp /etc/sysctl.conf root@:/etc/sysctl.conf

3- Apply the new configuration on the affected node:

 /usr/sbin/sysctl -p

4- Ensure that the command completes successfully without any "invalid syntax" errors.

5- Once the above steps are completed and verified, re-initiate the firmware update process using the Resource Group.

Impacted Versions 

PFxM 4.5.1.1 and lower

Fixed In Version

PFxM 4.5.2