Start a Conversation

Unsolved

This post is more than 5 years old

3626

February 15th, 2016 02:00

SDC drv_cfg --query_guid causes kernel panic on CentOS 7

Steps to reproduce:

  • Install CentOS 7 (minimal install)
  • Execute yum update
  • Compile and install ixgbe 4.3.13 (needed for Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T)
  • Install ScaleIO using the gateway
  • Execute /opt/emc/scaleio/sdc/bin/drv_cfg --query_guid on any SDC

output of kdump crash:

KERNEL: /usr/lib/debug/lib/modules/3.10.0-327.4.5.el7.x86_64/vmlinux

    DUMPFILE: /var/crash/127.0.0.1-2016-02-12-17:41:36/vmcore  [PARTIAL DUMP]

        CPUS: 8

        DATE: Fri Feb 12 17:41:23 2016

      UPTIME: 00:12:39

LOAD AVERAGE: 1.21, 1.10, 0.69

       TASKS: 199

    NODENAME: node2

     RELEASE: 3.10.0-327.4.5.el7.x86_64

     VERSION: #1 SMP Mon Jan 25 22:07:14 UTC 2016

     MACHINE: x86_64  (2200 Mhz)

      MEMORY: 31.9 GB

       PANIC: "BUG: unable to handle kernel paging request at 00007fff3e4b8570"

         PID: 3366

     COMMAND: "drv_cfg"

        TASK: ffff88084d073980  [THREAD_INFO: ffff880836604000]

         CPU: 7

       STATE: TASK_RUNNING (PANIC)

output of lspci:

00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 02)

00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 02)

00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 02)

00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 02)

00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 02)

00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 02)

00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 02)

00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 02)

00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 02)

00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)

00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)

00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04)

00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)

00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)

00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)

00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)

00:1f.0 ISA bridge: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller (rev 05)

00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)

00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)

02:00.0 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 0

02:00.1 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 1

02:00.2 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 2

02:00.3 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 3

03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T

03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T

06:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)

07:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

ff:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 02)

ff:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 02)

ff:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 02)

ff:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 02)

ff:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 02)

ff:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 02)

ff:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 02)

ff:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 02)

ff:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 02)

ff:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 02)

ff:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 02)

ff:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 02)

ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 02)

ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 02)

ff:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 02)

ff:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 02)

ff:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 02)

ff:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 02)

ff:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 02)

ff:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 02)

ff:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 02)

ff:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 02)

ff:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 02)

ff:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 02)

ff:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 02)

ff:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 02)

ff:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 02)

ff:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 02)

ff:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 02)

ff:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 02)

ff:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 02)

ff:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 02)

ff:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 02)

ff:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 02)

I have been able to reproduce it several times and I'm currently out of ideas to make it work.

Thanks a lot in advance for any help and best regards,

Luis Semprun

60 Posts

February 15th, 2016 07:00

can you please post the output of:

1. uname -a

2. cat /etc/release

5 Posts

February 15th, 2016 23:00

uname -a

Linux node2 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


cat /etc/centos-release

CentOS Linux release 7.2.1511 (Core)

60 Posts

February 16th, 2016 04:00

Unfortunately, CentOS 7.2 is not yet supported.

Please downgrade to CentOS 7.1 and make sure SELinux is not installed.

5 Posts

February 18th, 2016 04:00

Hi,


Unfortunately, it also crashes on CentOS 7.1.


uname -a

Linux node1 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/centos-release

CentOS Linux release 7.1.1503 (Core)

getenforce

Disabled

Mounting, reading & writing operations do work as expected (so far). My guess is that there's some kind of bug with the drv_cfg tool when installed on bare metal (as it works when the OS is virtualized).

Any ideas?

51 Posts

February 18th, 2016 14:00

semprunl,

I just tested this on an (admittedly different, but similar) virtual machine at both 7.1.1503 as well as 7.2.1511.

I am unable to get drv_cfg to error in this way, which jives with your experience of "if virtualized, no issue".

Do you have successful volume access otherwise? Is it only --query_guid that it crashes on, or any drv_cfg command?

Did you check the md5sum of the RHEL7 installer archive or re-download and extract on other boxes, md5summing there to verify it's not just a faulty copy of the SDC binary?

Since it works virtualized I'd investigate the difference in driver modules running between bare metal and virtual installs, and test with/without them.


The fact that you need to compile the ethernet driver also doesn't comfort me, and I wonder if the kernel is 100% happy with the NIC.

A non-answer, I know, but let me know what you find.

5 Posts

February 19th, 2016 03:00

Hi Rush,

I do have successful volume access but the following drv_cfg commands are crashing the kernel:

--query_diag_counters

--query_guid


The md5sum is valid and the compiled driver is the most current release of the ixgbe driver for Linux, which supports kernel versions 2.6.18 up through 4.3.3.  It also has been tested on the following distributions:

  - RHEL 6.7

  - RHEL 7.2

  - SLES 11SP4

  - SLES 12PS1

I also tried with an older version of ixgbe (4.1.2) which was tested with RHEL 7.1 but still the same outcome (kernel crash).

Thanks for taking a look.

5 Posts

February 19th, 2016 03:00

You may find useful the fact that I can not reproduce the crash on CentOS 6.7 using the same server and with the same Ethernet driver.

5 Practitioner

 • 

274.2K Posts

April 11th, 2016 23:00

Hi Semprunl,

Could you solve the problem? I also got a kernel panic with Centos 7.2 and ScaleIO 1.32.3, 1.32.4, 2.0. All panics are related to the sdc. I could reproduce the panic with 3 machines. All running Centos bare metal. I'm testing on 3 Intel NUCs

output of lspci:

00:00.0 Host bridge: Intel Corporation Broadwell-U Host Bridge -OPI (rev 09)

00:02.0 VGA compatible controller: Intel Corporation Broadwell-U Integrated Graphics (rev 09)

00:03.0 Audio device: Intel Corporation Broadwell-U Audio Controller (rev 09)

00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI Controller (rev 03)

00:16.0 Communication controller: Intel Corporation Wildcat Point-LP MEI Controller #1 (rev 03)

00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (3) I218-V (rev 03)

00:1b.0 Audio device: Intel Corporation Wildcat Point-LP High Definition Audio Controller (rev 03)

00:1c.0 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #1 (rev e3)

00:1c.3 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #4 (rev e3)

00:1d.0 USB controller: Intel Corporation Wildcat Point-LP USB EHCI Controller (rev 03)

00:1f.0 ISA bridge: Intel Corporation Wildcat Point-LP LPC Controller (rev 03)

00:1f.2 SATA controller: Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode] (rev 03)

00:1f.3 SMBus: Intel Corporation Wildcat Point-LP SMBus Controller (rev 03)

02:00.0 Network controller: Intel Corporation Wireless 7265 (rev 59)

kdump:

[  653.630233] BUG: unable to handle kernel paging request at 00007ffd5b2d5e20

[  653.630282] IP: [ ] ioctl_Handler+0x8bd/0xda0 [scini]

[  653.630333] PGD 43a808067 PUD 43afa1067 PMD 43afa3067 PTE 800000042a9a3067

[  653.630382] Oops: 0003 [#1] SMP

[  653.630403] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver scini(POE) nfs fscache intel_powerclamp coretemp intel_rapl kvm_intel kvm arc4 crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi iTCO_wdt iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic iwlmvm mac80211 snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep aesni_intel lrw snd_seq gf128mul snd_seq_device glue_helper ablk_helper cryptd sg iwlwifi snd_pcm ir_lirc_codec lirc_dev ir_sony_decoder ir_sanyo_decoder ir_mce_kbd_decoder pcspkr btusb ir_jvc_decoder cfg80211 bluetooth ir_rc6_decoder rfkill ir_nec_decoder ir_rc5_decoder snd_timer snd mei_me soundcore mei i2c_i801 lpc_ich shpchp mfd_core rc_rc6_mce nuvoton_cir rc_core acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif

[  653.630740]  crct10dif_generic i915 ahci libahci crct10dif_pclmul crct10dif_common crc32c_intel i2c_algo_bit libata drm_kms_helper serio_raw e1000e drm ptp pps_core sdhci_acpi sdhci video mmc_core i2c_hid i2c_core dm_mirror dm_region_hash dm_log dm_mod

[  653.630831] CPU: 2 PID: 11545 Comm: drv_cfg Tainted: P           OE  ------------   3.10.0-327.13.1.el7.x86_64 #1

[  653.630863] Hardware name: TAROX ECO 44 G5/NUC5i5RYB, BIOS RYBDWi35.86A.0355.2016.0224.1501 02/24/2016

[  653.630894] task: ffff8800974bc500 ti: ffff88043a81c000 task.ti: ffff88043a81c000

[  653.630918] RIP: 0010:[ ]  [ ] ioctl_Handler+0x8bd/0xda0 [scini]

[  653.630950] RSP: 0018:ffff88043a81fe20  EFLAGS: 00010246

[  653.630970] RAX: 00000000c30b5096 RBX: 00007ffd5b2d5e10 RCX: 0000000000000000

[  653.630993] RDX: 0000000000000018 RSI: ffff88043a81fe50 RDI: 00007ffd5b2d5e28

[  653.631015] RBP: ffff88043a81feb0 R08: 0000000000000000 R09: 0000000000000000

[  653.631039] R10: 00007ffd5b2d5b80 R11: 0000000000000000 R12: 0000000000000000

[  653.631061] R13: 00007ffd5b2d5e10 R14: 00007ffd5b2d5e10 R15: 0000000000000000

[  653.631084] FS:  00007f737ef28780(0000) GS:ffff880456d00000(0000) knlGS:0000000000000000

[  653.631111] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[  653.631130] CR2: 00007ffd5b2d5e20 CR3: 000000043a80a000 CR4: 00000000003407e0

[  653.631152] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

[  653.631175] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

[  653.631198] Stack:

[  653.631205]  ffff880406e04b00 c24410ced1ad23a3 1f2562d7c30b5096 c24410ced1ad23a3

[  653.631232]  1f2562d7c30b5096 ffffffff811efeeb ffff880440159f20 ffff88043a81fec8

[  653.631261]  000fffff6e6963dc fffdffef00000000 0000000000000001 000000001c8a8a4f

[  653.631288] Call Trace:

[  653.631299]  [ ] ? do_filp_open+0x4b/0xb0

[  653.631318]  [ ] do_vfs_ioctl+0x2e5/0x4c0

[  653.631340]  [ ] ? file_has_perm+0xae/0xc0

[  653.631360]  [ ] ? __do_page_fault+0x91/0x450

[  653.631379]  [ ] SyS_ioctl+0xa1/0xc0

[  653.631399]  [ ] system_call_fastpath+0x16/0x1b

[  653.631418] Code: 18 00 00 00 48 89 df 48 89 44 24 18 48 8b 44 24 10 48 89 44 24 20 e8 a3 21 96 e0 85 c0 49 89 c4 0f 85 bf 02 00 00 e8 53 71 ff ff <89> 43 10 45 31 e4 e8 68 71 ff ff 89 43 14 e9 d0 f7 ff ff 48 8d

[  653.631538] RIP  [ ] ioctl_Handler+0x8bd/0xda0 [scini]

[  653.631566]  RSP

[  653.631577] CR2: 00007ffd5b2d5e20

April 21st, 2016 05:00

Can you please supply the get_info output (including the crash dump, if you have it)?

Either from run collect logs from the IM-web -> maintenance view, or run /opt/emc/scaleio/mdm/bin/get_info.sh

April 21st, 2016 05:00

Can you please supply the get_info output (including the crash dump, if you have it)?

Either from run collect logs from the IM-web -> maintenance view, or run/opt/emc/scaleio/mdm/bin/get_info.sh

306 Posts

April 22nd, 2016 05:00

Hi Tomer,

We will try to reproduce it in the lab and escalate to L3 if necessary.

Thank you,

Pawel

5 Practitioner

 • 

274.2K Posts

July 13th, 2016 17:00

I am running into this issue without having to follow as many of the steps.

I've tried this on 2 systems, one Broadwell-based XeonD 1541 and one Skylake-based 6700k.

I've tested CentOS 7.2 & Ubuntu 14.04 with the 5014 sdc and the 6035 sdc. The result is the same for both.

My steps to reproduce (for CentOS):

install CentOS 7.2

yum update -y

reboot

yum install -y numactl libaio

rpm -U ScaleIO-sdc-2.0-6035.0.el7.x86_64.rpm

during the install, the scini driver attempts to start and this leads to a kernel panic.

because of the above details, I even forcefully disabled / unloaded both the ixgbe and igb network devices (only network devices on my system) and had the same result (kernel panic)

i will try with centos 6.7 next. i also tried with a virtual machine on top and had no issues.

lspci:

00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 03)

00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 03)

00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03)

00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03)

00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03)

00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 03)

00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 03)

00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 03)

00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 03)

00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)

00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)

00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04)

00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)

00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)

00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)

00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)

00:1f.0 ISA bridge: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller (rev 05)

00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)

00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01)

02:00.0 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 0

02:00.1 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 1

02:00.2 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 2

02:00.3 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 3

03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T

03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T

05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

06:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)

07:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

ff:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)

ff:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)

ff:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)

ff:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 03)

ff:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0c.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0c.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0c.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0c.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)

ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 03)

ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 03)

ff:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)

ff:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)

ff:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)

ff:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 03)

ff:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 03)

ff:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 03)

ff:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 03)

ff:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 03)

ff:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 03)

ff:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 03)

ff:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 03)

ff:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 03)

ff:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 03)

ff:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control (rev 03)

ff:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control (rev 03)

ff:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 03)

ff:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 03)

ff:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)

ff:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)

ff:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)

ff:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)

ff:15.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control (rev 03)

ff:15.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control (rev 03)

ff:15.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error (rev 03)

ff:15.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error (rev 03)

ff:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)

ff:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)

ff:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)

ff:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)

ff:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)

ff:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)

ff:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)

message from crash:

[  105.126495] scini: module license 'Proprietary' taints kernel.

[  105.126499] Disabling lock debugging due to kernel taint

[  105.126839] scini: module verification failed: signature and/or required key missing - tainting kernel

[  105.127534] scini: applying kernel_stack fix up

[  105.129741] ScaleIO R2_0 drvMain_Init:61 :Driver starting load process. HZ 1000...

[  105.132868] ScaleIO R2_0 mosCpu_Init:80 :eax=00000000, ebx=00000000, ecx=00000000, edx=00000000

[  105.134555] ScaleIO R2_0 mosT10Dif_Init:705 :(T10DIF) T10DIF layer using CPU (PCLMUL) acceleration !!!

[  105.136316] ScaleIO R2_0 drvMain_Init:63 :Load process done. Status SUCCESS

[  105.147009] ScaleIO R2_0 ioctl_HandleSetIniGuid:552 :Set GUID (E587E4E5-5A56-4C3E-A18B-0C06EEA51F2D) finished . status 65 (SUCCESS)

[  105.152864] BUG: unable to handle kernel paging request at 00007ffe7a969188

[  105.154520] IP: [ ] ioctl_Handler+0x890/0x1e00 [scini]

[  105.156184] PGD 1024f5f067 PUD 10275aa067 PMD 1024210067 PTE 101315c067

[  105.157847] Oops: 0001 [#1] SMP

[  105.159507] Modules linked in: scini(POE) intel_powerclamp coretemp intel_rapl kvm_intel kvm iTCO_wdt crc32_pclmul iTCO_vendor_support mxm_wmi vfat ghash_clmulni_intel fat aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_usb_audio ipmi_ssif snd_usbmidi_lib snd_hwdep snd_rawmidi snd_seq snd_seq_device snd_pcm sb_edac pcspkr edac_core i2c_i801 snd_timer snd soundcore lpc_ich mfd_core mei_me mei shpchp ipmi_si ipmi_msghandler wmi acpi_pad ip_tables xfs libcrc32c ast syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ahci crct10dif_pclmul libahci crct10dif_common drm libata crc32c_intel i2c_core nvme

[  105.164973] CPU: 11 PID: 2361 Comm: drv_cfg Tainted: P           OE  ------------   3.10.0-327.22.2.el7.x86_64 #1

[  105.166695] Hardware name: Supermicro Super Server/X10SDV-TLN4F, BIOS 1.1 03/02/2016

[  105.168364] task: ffff8810202e3980 ti: ffff8810121b0000 task.ti: ffff8810121b0000

[  105.170039] RIP: 0010:[ ]  [ ] ioctl_Handler+0x890/0x1e00 [scini]

[  105.171743] RSP: 0018:ffff8810121b3d48  EFLAGS: 00010246

[  105.173443] RAX: 0000000000000000 RBX: ffff8810267cca00 RCX: 00007ffe7a969180

[  105.175092] RDX: 00007ffe7a969180 RSI: 0000000000006107 RDI: ffff8810267cca00

[  105.176681] RBP: ffff8810121b3eb0 R08: 0000000000000000 R09: 0000000000000000

[  105.178281] R10: 00007ffe7a968c70 R11: 0000000000000000 R12: 00007ffe7a969180

[  105.179886] R13: 00007ffe7a969180 R14: 00007ffe7a969180 R15: 0000000000000001

[  105.181497] FS:  00007f24282f4800(0000) GS:ffff88107fd60000(0000) knlGS:0000000000000000

[  105.183125] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[  105.184766] CR2: 00007ffe7a969188 CR3: 000000102720e000 CR4: 00000000003407e0

[  105.186425] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

[  105.188087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

[  105.189739] Stack:

[  105.191369]  ffff8810121b3de4 ffff88103ec0a000 ffff8810267cca00 ffff880000000026

[  105.193024]  ffff881023957b00 ffff88102710ed80 00000000d892704b ffff8810121b3e50

[  105.194677]  ffff8810267cca00 ffff8810286d9000 ffff8810121b3f28 ffff8810202e3980

[  105.196364] Call Trace:

[  105.197996]  [ ] ? path_openat+0xc2/0x490

[  105.199647]  [ ] ? do_filp_open+0x4b/0xb0

[  105.201286]  [ ] do_vfs_ioctl+0x2e5/0x4c0

[  105.202909]  [ ] ? file_has_perm+0xae/0xc0

[  105.204523]  [ ] SyS_ioctl+0xa1/0xc0

[  105.206134]  [ ] system_call_fastpath+0x16/0x1b

[  105.207740] Code: c0 0f 85 4f 0b 00 00 8b bd 10 ff ff ff e8 09 c3 01 00 83 bd 10 ff ff ff 41 0f 85 ec f9 ff ff f7 d8 e9 ef f9 ff ff 0f 1f 44 00 00 <48> 8b 5a 08 bf 2f 00 00 00 85 db 0f 84 45 06 00 00 81 fb 30 75

[  105.209551] RIP  [ ] ioctl_Handler+0x890/0x1e00 [scini]

[  105.211250]  RSP

[  105.212930] CR2: 00007ffe7a969188

5 Practitioner

 • 

274.2K Posts

July 13th, 2016 20:00

I tested CentOS 6.x as @semprunl did and the sdc packaged installed without issues. I also mounted a volume, formatted it, and wrote to / read from it without any issues.

22 Posts

September 22nd, 2016 01:00

Hi, was anyone able to solve kernel crash problem ? We ran into the same issue. And we found out, that crash is related to Intel Xeon E5 v4 CPU ( on v3 SDC works fine).

Thanks,

Matas

22 Posts

September 22nd, 2016 05:00

HI, and seems we found the solution:   SDC version: EMC-ScaleIO-sdc-2.0-7120.0.el7.x86_64.rpm  doesn't crash kernel.

Our setup: CentOS 3.10.0-327.36.1.el7.x86_64

Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 01)

Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+

No Events found!

Top