PowerFlex 3.6: Red Hat OpenStack component deploy paused

Summary: Red Hat OpenStack Platform (RHOSP) deployment is paused on an SDC node due to service enable failure. If retried it is successful. Assumed to be when port flapping is enabled

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Scenario

Deployment of a new RHOSP environment using Ansible script which installs a new PowerFlex 3.6.x SDC.

The neutron-cleanup daemon fails to start. The time of failure is correlated with port flapping feature events.

 

Symptoms

ansible.log - failed to start the neutron-cleanup daemon:

2022-02-25 12:51:20,251 p=784155 u=mistral n=ansible | host: clouddeva-compute-std-9, task: None
2022-02-25 12:51:20,253 p=784155 u=mistral n=ansible | 2022-02-25 12:51:20.253296 | 5254002b-b028-26d5-f46a-000000005317 |      FATAL | Enabling the cleanup service | clouddeva-compute-std-3 | error={"changed": false, "msg": "Unable to enable service neutron-cleanup: Created symlink /etc/systemd/system/multi-user.target.wants/neutron-cleanup.service → /usr/lib/systemd/system/neutron-cleanup.service.\nFailed to reload daemon: Connection timed out\n"}
2022-02-25 12:51:20,315 p=784155 u=mistral n=ansible | host: clouddeva-controller-0, task: None

SDC /var/log/messages - failed daemon start

Feb 25 07:48:15 clouddeva-compute-std-3 kernel: PowerFlex netPortFlap_PrepareUpdateMessage:164 :FLAP: Prepared report with 9 (out of 9) disqualified and 7 (out of 13) weak violating portals (unstable)
Feb 25 07:48:15 clouddeva-compuFeb 25 07:48:16 clouddeva-compute-std-3 ansible-stat[24479]: Invoked with path=/etc/modules-load.d/openvswitch.conf follow=False get_checksum=True checksum_algorithm=sha1 get_md5=False get_mime=True get_attributes=True
Feb 25 07:48:17 clouddeva-compute-ste-std-3 kernel: [4295286250] Connected to SDS with ID 37bb9ae500000009
Feb 25 07:48:15 clouddeva-compute-std-3 kernel: [4295286255] Disconnected from SDS with ID 37bb9ae600000008
Feb 25 07:48:15 clouddeva-compute-std-3 kernel: [4295286261] Disconnected from SDS with ID 37bb9adf00000003
Feb 25 07:48:15 clouddeva-compute-std-3 kernel: [4295286328] Connected to SDS with ID 37bb9ae000000004
Feb 25 07:48:15 clouddeva-compute-std-3 ansible-stat[24380]: Invoked with path=/etc/modules follow=False get_md5=False get_checksum=True get_mime=True get_attributes=True checksum_algorithm=sha1
Feb 25 07:48:15 clouddeva-compute-std-3 kernel: [4295286356] Connected to SDS with ID 37bb9ae600000008
Feb 25 07:48:16 clouddeva-compute-std-3 ansible-modprobe[24424]: Invoked with name=openvswitch state=present params=
Feb 25 07:48:16 clouddeva-compute-std-3 kernel: [4295286563] Connected to SDS with ID 37bb9adf00000003
Feb 25 07:48:16 clouddeva-compute-std-3 kernel: [4295286850] Connected to SDS with ID 37bb9add00000001
td-3 ansible-copy[24523]: Invoked with dest=/etc/modules-load.d/openvswitch.conf src=/home/tripleo-admin/.ansible/tmp/ansible-tmp-1645793296.6471314-801998-190517710088290/source mode=None follow=False _original_basename=module-load.conf.j2 checksum=aca7c8bb7013e03e6cf531689fb6220d9e469d63 backup=False force=True unsafe_writes=False content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None regexp=None delimiter=None
Feb 25 07:48:18 clouddeva-compute-std-3 ansible-stat[24619]: Invoked with path=/usr/libexec/neutron-cleanup follow=False get_checksum=True checksum_algorithm=sha1 get_md5=False get_mime=True get_attributes=True
Feb 25 07:48:18 clouddeva-compute-std-3 ansible-copy[24693]: Invoked with dest=/usr/libexec/neutron-cleanup force=True mode=0755 src=/home/tripleo-admin/.ansible/tmp/ansible-tmp-1645793298.1814697-802218-228907330848454/source _original_basename=tmpmhvsxu9b follow=False checksum=a9feff7c7567b3ae8baa2006455b8cc8b370d239 backup=False unsafe_writes=False content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None regexp=None delimiter=None
Feb 25 07:48:19 clouddeva-compute-std-3 ansible-stat[24759]: Invoked with path=/usr/lib/systemd/system/neutron-cleanup.service follow=False get_checksum=True checksum_algorithm=sha1 get_md5=False get_mime=True get_attributes=True
Feb 25 07:48:19 clouddeva-compute-std-3 kernel: PowerFlex netCon_IsKaNeeded:4787 :CON 0000000058dfd327 peer 10.232.248.5 didn't receive message for 30 iterations.  Marking as down
Feb 25 07:48:19 clouddeva-compute-std-3 ansible-copy[24803]: Invoked with dest=/usr/lib/systemd/system/neutron-cleanup.service force=True src=/home/tripleo-admin/.ansible/tmp/ansible-tmp-1645793299.0068667-802347-201781290304692/source _original_basename=tmpx3w9vtsi follow=False checksum=6b6cd9f074903a28d054eb530a10c7235d0c39fc backup=False unsafe_writes=False content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None regexp=None delimiter=None
Feb 25 07:48:20 clouddeva-compute-std-3 ansible-systemd[24867]: Invoked with enabled=True name=neutron-cleanup daemon_reload=False daemon_reexec=False no_block=False state=None force=None masked=None user=None scope=None
Feb 25 07:48:20 clouddeva-compute-std-3 systemd[1]: Reloading.

SDC /var/log/messages - successful daemon start

Mar  2 12:25:43 clouddeva-compute-std-0 ansible-modprobe[36359]: Invoked with name=openvswitch state=present params=
Mar  2 12:25:43 clouddeva-compute-std-0 ansible-stat[36411]: Invoked with path=/etc/modules-load.d/openvswitch.conf follow=False get_checksum=True checksum_algorithm=sha1 get_md5=False get_mime=True get_attributes=True
Mar  2 12:25:44 clouddeva-compute-std-0 ansible-copy[36472]: Invoked with dest=/etc/modules-load.d/openvswitch.conf src=/home/tripleo-admin/.ansible/tmp/ansible-tmp-1646241943.6915667-45654-245764505198253/source mode=None follow=False _original_basename=module-load.conf.j2 checksum=aca7c8bb7013e03e6cf531689fb6220d9e469d63 backup=False force=True unsafe_writes=False content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None regexp=None delimiter=None
Mar  2 12:25:45 clouddeva-compute-std-0 ansible-stat[36582]: Invoked with path=/usr/libexec/neutron-cleanup follow=False get_checksum=True checksum_algorithm=sha1 get_md5=False get_mime=True get_attributes=True
Mar  2 12:25:45 clouddeva-compute-std-0 ansible-copy[36628]: Invoked with dest=/usr/libexec/neutron-cleanup force=True mode=0755 src=/home/tripleo-admin/.ansible/tmp/ansible-tmp-1646241945.256886-45861-233404909922568/source _original_basename=tmpce1pezaw follow=False checksum=a9feff7c7567b3ae8baa2006455b8cc8b370d239 backup=False unsafe_writes=False content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None regexp=None delimiter=None
Mar  2 12:25:46 clouddeva-compute-std-0 ansible-stat[36723]: Invoked with path=/usr/lib/systemd/system/neutron-cleanup.service follow=False get_checksum=True checksum_algorithm=sha1 get_md5=False get_mime=True get_attributes=True
Mar  2 12:25:46 clouddeva-compute-std-0 ansible-copy[36767]: Invoked with dest=/usr/lib/systemd/system/neutron-cleanup.service force=True src=/home/tripleo-admin/.ansible/tmp/ansible-tmp-1646241946.3689635-46015-253217383764482/source _original_basename=tmphwqutgs0 follow=False checksum=6b6cd9f074903a28d054eb530a10c7235d0c39fc backup=False unsafe_writes=False content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None regexp=None delimiter=None
Mar  2 12:25:47 clouddeva-compute-std-0 ansible-systemd[36808]: Invoked with enabled=True name=neutron-cleanup daemon_reload=False daemon_reexec=False no_block=False state=None force=None masked=None user=None scope=None
Mar  2 12:25:47 clouddeva-compute-std-0 systemd[1]: Reloading.

 

Impact

RHOSP deployment paused and must be retried to proceed.

Cause

Unknown

Resolution

One can use any of the following workarounds to resolve the paused state:

  1. Proceed with retry.
  2. Disable port flapping (as part of the automated process) once the SDC/s are registered in PowerFlex, by running the commands below.

Disable port flapping for a specific SDC:

 scli --set_performance_parameters --sdc_id  --tech --sdc_port_flap_feature_enabled 0

Disable port flapping for all SDCs:

 scli --set_performance_parameters --all_sdc --tech --sdc_port_flap_feature_enabled 0

 

Impacted Versions

PowerFlex 3.6 and higher

Affected Products

PowerFlex rack, ScaleIO
Article Properties
Article Number: 000269490
Article Type: Solution
Last Modified: 28 Jul 2025
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.