FalkAlexander
1 Copper

XPS 13 9370: ath10k_pci firmware crash

The wifi firmware crashes often when using a openvpn connection. Then there isn't a network connection or reconnect anymore possible. When trying to reconnect the complete system freezes and a hard reset is needed.

This happens on all linux distributions I have tested so far. Ubuntu 18.04, Fedora 28, Arch Linux.

Kernel log/stack trace: https://paste.gnome.org/pjmxbypjg/1brna1

Kernel version: Linux xps 4.17.6-1-ARCH #1 SMP PREEMPT Wed Jul 11 19:14:29 UTC 2018 x86_64 GNU/Linux

Kernel boot commandline: 

rd.luks.name=<uuid>=cryptlvm root=/dev/mapper/main-root rw resume=/dev/mapper/main-swap i915.enable_fbc=1 i915.enable_guc=3 i915.fastboot=1 i915.enable_psr=2 pcie_aspm.policy=powersave mem_sleep_default=deep

 

 

12 Replies
Venemo
1 Nickel

Re: XPS 13 9370: ath10k_pci firmware crash

Have you tried to update the firmware?  New firmware can be found here: https://github.com/kvalo/ath10k-firmware usually it takes a while before it gets into the linux-firmware packages of distributions. This is how you can update it: https://www.dell.com/support/article/us/en/19/sln306440/killer-n1535-wireless-firmware-manual-update...

0 Kudos
FalkAlexander
1 Copper

Re: XPS 13 9370: ath10k_pci firmware crash

I've tried this already and also compared the checksum of linux-firmware ath10k binary blobs from Arch Linux with the ones from the git repository. They're the same.

Meanwhile a firmware crash occurred again. But the kernel was able to recover instead crashing.

Jul 19 14:06:25 xps kernel: ath10k_pci 0000:01:00.0: device successfully recovered
0 Kudos
Intro
1 Nickel

Re: XPS 13 9370: ath10k_pci firmware crash

I'm suffering the same issue even without using VPN, and it's absolutely 

Every 5-10 minutes (or never in other sessions), my connection drops while still connected. I mean, the laptop is still connected as per iwconfig, but communication is broken. It's a firmware crash as you can see:

[  820.463814] ath10k_pci 0000:02:00.0: HTC Rx: invalid eid 14
[  825.825910] ath10k_pci 0000:02:00.0: firmware crashed! (guid c8e34a2a-a75b-42ef-aa38-aeb186214e45)
[  825.825928] ath10k_pci 0000:02:00.0: qca6174 hw3.2 target 0x05030000 chip_id 0x00340aff sub 1a56:143a
[  825.825934] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 1 tracing 1 dfs 0 testmode 0
[  825.827589] ath10k_pci 0000:02:00.0: firmware ver WLAN.RM.4.4.1-00119-QCARMSWP-1 api 6 features wowlan,ignore-otp crc32 859225d1
[  825.828859] ath10k_pci 0000:02:00.0: board_file api 2 bmi_id N/A crc32 20d869c3
[  825.828867] ath10k_pci 0000:02:00.0: htt-ver 3.56 wmi-op 4 htt-op 3 cal otp max-sta 32 raw 0 hwcrypto 1
[  825.830881] ath10k_pci 0000:02:00.0: firmware register dump:
[  825.830891] ath10k_pci 0000:02:00.0: [00]: 0x05030000 0x000015B3 0x009FCE17 0x00955B31
[  825.830899] ath10k_pci 0000:02:00.0: [04]: 0x009FCE17 0x00060730 0x00000005 0xFFFC0000
[  825.830906] ath10k_pci 0000:02:00.0: [08]: 0x00000001 0x000CD21C 0x000CE5A4 0x000CCABD
[  825.830914] ath10k_pci 0000:02:00.0: [12]: 0x00000009 0x00000001 0x009DC9B0 0x009DC9BA
[  825.830920] ath10k_pci 0000:02:00.0: [16]: 0x00952CC4 0x009392F3 0x00000000 0x0091080D
[  825.830927] ath10k_pci 0000:02:00.0: [20]: 0x409FCE17 0x0040E848 0x000CD1A4 0x00000000
[  825.830934] ath10k_pci 0000:02:00.0: [24]: 0x80A03408 0x0040E8A8 0x00000014 0xC09FCE17
[  825.830941] ath10k_pci 0000:02:00.0: [28]: 0x80A03618 0x0040E8C8 0x00000014 0x00001000
[  825.830948] ath10k_pci 0000:02:00.0: [32]: 0x80A036A6 0x0040E8F8 0x000CD497 0x0000FE00
[  825.830955] ath10k_pci 0000:02:00.0: [36]: 0x80939AB1 0x0040E918 0x00407CC8 0x00000600
[  825.830962] ath10k_pci 0000:02:00.0: [40]: 0x80A03498 0x0040E958 0x00407C78 0x00407C7C
[  825.830969] ath10k_pci 0000:02:00.0: [44]: 0x80926FBB 0x0040E998 0x00000000 0x00000000
[  825.830976] ath10k_pci 0000:02:00.0: [48]: 0x809E34B9 0x0040E9C8 0x0040A054 0x00000000
[  825.830983] ath10k_pci 0000:02:00.0: [52]: 0x809E378B 0x0040EA38 0x0040A054 0x00000001
[  825.830990] ath10k_pci 0000:02:00.0: [56]: 0x80932EF9 0x0040EA68 0x0040A054 0x00428E80
[  825.830995] ath10k_pci 0000:02:00.0: Copy Engine register dump:
[  825.831011] ath10k_pci 0000:02:00.0: [00]: 0x00034400   8   8   3   3
[  825.831026] ath10k_pci 0000:02:00.0: [01]: 0x00034800   8   8 309 310
[  825.831041] ath10k_pci 0000:02:00.0: [02]: 0x00034c00  26  26  89  90
[  825.831055] ath10k_pci 0000:02:00.0: [03]: 0x00035000  19  19  21  19
[  825.831069] ath10k_pci 0000:02:00.0: [04]: 0x00035400 5405 5405 208 144
[  825.831084] ath10k_pci 0000:02:00.0: [05]: 0x00035800   0   0  64   0
[  825.831098] ath10k_pci 0000:02:00.0: [06]: 0x00035c00  29  29  22  22
[  825.831113] ath10k_pci 0000:02:00.0: [07]: 0x00036000   1   1   1   1
[  826.655807] ath10k_pci 0000:02:00.0: Unknown eventid: 118809
[  826.658810] ath10k_pci 0000:02:00.0: Unknown eventid: 90118
[  826.743240] ath10k_pci 0000:02:00.0: device successfully recovered

Regardless of the final message,

device successfully recovered

the connection is not resumed.

The only way I found to get it back is restarting the service

 

service network-manager restart

which takes serveral seconds, even minutes. Restarting the laptop does not get the connection back sometimes. Once restarted, the log looks like

 

 

[    3.396515] ath10k_pci 0000:02:00.0: enabling device (0000 -> 0002)
[    3.398574] ath10k_pci 0000:02:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[    3.688041] ath10k_pci 0000:02:00.0: qca6174 hw3.2 target 0x05030000 chip_id 0x00340aff sub 1a56:143a
[    3.688045] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 1 tracing 1 dfs 0 testmode 0
[    3.688584] ath10k_pci 0000:02:00.0: firmware ver WLAN.RM.4.4.1-00119-QCARMSWP-1 api 6 features wowlan,ignore-otp crc32 859225d1
[    3.754161] ath10k_pci 0000:02:00.0: board_file api 2 bmi_id N/A crc32 20d869c3
[    4.341127] ath10k_pci 0000:02:00.0: Unknown eventid: 118809
[    4.344132] ath10k_pci 0000:02:00.0: Unknown eventid: 90118
[    4.344710] ath10k_pci 0000:02:00.0: htt-ver 3.56 wmi-op 4 htt-op 3 cal otp max-sta 32 raw 0 hwcrypto 1
[    4.443957] ath10k_pci 0000:02:00.0 wlp2s0: renamed from wlan0
[    5.355614] ath10k_pci 0000:02:00.0: Unknown eventid: 118809
[    5.358611] ath10k_pci 0000:02:00.0: Unknown eventid: 90118

...not related stuff...

[  820.463814] ath10k_pci 0000:02:00.0: HTC Rx: invalid eid 14
[  825.825910] ath10k_pci 0000:02:00.0: firmware crashed! (guid c8e34a2a-a75b-42ef-aa38-aeb186214e45)
[  825.825928] ath10k_pci 0000:02:00.0: qca6174 hw3.2 target 0x05030000 chip_id 0x00340aff sub 1a56:143a
[  825.825934] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 1 tracing 1 dfs 0 testmode 0
[  825.827589] ath10k_pci 0000:02:00.0: firmware ver WLAN.RM.4.4.1-00119-QCARMSWP-1 api 6 features wowlan,ignore-otp crc32 859225d1
[  825.828859] ath10k_pci 0000:02:00.0: board_file api 2 bmi_id N/A crc32 20d869c3
[  825.828867] ath10k_pci 0000:02:00.0: htt-ver 3.56 wmi-op 4 htt-op 3 cal otp max-sta 32 raw 0 hwcrypto 1
[  825.830881] ath10k_pci 0000:02:00.0: firmware register dump:
[  825.830891] ath10k_pci 0000:02:00.0: [00]: 0x05030000 0x000015B3 0x009FCE17 0x00955B31
[  825.830899] ath10k_pci 0000:02:00.0: [04]: 0x009FCE17 0x00060730 0x00000005 0xFFFC0000
[  825.830906] ath10k_pci 0000:02:00.0: [08]: 0x00000001 0x000CD21C 0x000CE5A4 0x000CCABD
[  825.830914] ath10k_pci 0000:02:00.0: [12]: 0x00000009 0x00000001 0x009DC9B0 0x009DC9BA
[  825.830920] ath10k_pci 0000:02:00.0: [16]: 0x00952CC4 0x009392F3 0x00000000 0x0091080D
[  825.830927] ath10k_pci 0000:02:00.0: [20]: 0x409FCE17 0x0040E848 0x000CD1A4 0x00000000
[  825.830934] ath10k_pci 0000:02:00.0: [24]: 0x80A03408 0x0040E8A8 0x00000014 0xC09FCE17
[  825.830941] ath10k_pci 0000:02:00.0: [28]: 0x80A03618 0x0040E8C8 0x00000014 0x00001000
[  825.830948] ath10k_pci 0000:02:00.0: [32]: 0x80A036A6 0x0040E8F8 0x000CD497 0x0000FE00
[  825.830955] ath10k_pci 0000:02:00.0: [36]: 0x80939AB1 0x0040E918 0x00407CC8 0x00000600
[  825.830962] ath10k_pci 0000:02:00.0: [40]: 0x80A03498 0x0040E958 0x00407C78 0x00407C7C
[  825.830969] ath10k_pci 0000:02:00.0: [44]: 0x80926FBB 0x0040E998 0x00000000 0x00000000
[  825.830976] ath10k_pci 0000:02:00.0: [48]: 0x809E34B9 0x0040E9C8 0x0040A054 0x00000000
[  825.830983] ath10k_pci 0000:02:00.0: [52]: 0x809E378B 0x0040EA38 0x0040A054 0x00000001
[  825.830990] ath10k_pci 0000:02:00.0: [56]: 0x80932EF9 0x0040EA68 0x0040A054 0x00428E80
[  825.830995] ath10k_pci 0000:02:00.0: Copy Engine register dump:
[  825.831011] ath10k_pci 0000:02:00.0: [00]: 0x00034400   8   8   3   3
[  825.831026] ath10k_pci 0000:02:00.0: [01]: 0x00034800   8   8 309 310
[  825.831041] ath10k_pci 0000:02:00.0: [02]: 0x00034c00  26  26  89  90
[  825.831055] ath10k_pci 0000:02:00.0: [03]: 0x00035000  19  19  21  19
[  825.831069] ath10k_pci 0000:02:00.0: [04]: 0x00035400 5405 5405 208 144
[  825.831084] ath10k_pci 0000:02:00.0: [05]: 0x00035800   0   0  64   0
[  825.831098] ath10k_pci 0000:02:00.0: [06]: 0x00035c00  29  29  22  22
[  825.831113] ath10k_pci 0000:02:00.0: [07]: 0x00036000   1   1   1   1
[  826.655807] ath10k_pci 0000:02:00.0: Unknown eventid: 118809
[  826.658810] ath10k_pci 0000:02:00.0: Unknown eventid: 90118
[  826.743240] ath10k_pci 0000:02:00.0: device successfully recovered
[ 1067.396941] ath10k_pci 0000:02:00.0: failed to flush transmit queue (skip 0 ar-state 1): 0
[ 1072.516877] ath10k_pci 0000:02:00.0: failed to flush transmit queue (skip 0 ar-state 1): 0

 

As you can see, the firmware crashes again, but, magically, the connection is back regardless that. 

Might it be related to the hardware version? 9370 brings an HW3.2 version, but the available firmware is HW3.0

Testbed:

  • OS: Ubuntu from 16.04 t 18.04.1
  • Kernel: from 4.4.x to 4.18.x
  • Firmware: from the original one that brought Ubuntu 16.04 API4 (3 years old), to latest API6 (4.4.1-119, 2018-09-18) from the official source: https://github.com/kvalo/ath10k-firmware
  • WiFI power management ON and OFF. Nowadays is OFF.

 

As a side note, I've restarted the service twice during the time needed to write this post!!!

seansch00
1 Copper

Re: XPS 13 9370: ath10k_pci firmware crash

I've come across this issue as well on Ubuntu 18.10 using the default 4.18 kernel. When installing linux-image-4.15.0-34-generic from 18.04 the issue disappears. I've seen reference to the ath10k driver crashing on newer kernels, hopefully it is patched soon.

ShedG
1 Nickel

Re: XPS 13 9370: ath10k_pci firmware crash

This is how you can update it: https://www.dell.com/support/article/us/en/19/sln306440/killer-n1535-wireless-firmware-manual-update... (Link probably broken)

Those instructions refer to the Killer 1535 card. The 9370 uses the 1435 card. Are the instructions still appropriate?

 

tomxor
1 Nickel

Re: XPS 13 9370: ath10k_pci firmware crash

I'm having the same problems on Linux 4.18 with Debian. Some times i've gone weeks without an issue then it will hang every hour for the whole day to the point where I have to uninstall the **bleep** driver to get any work done.

0 Kudos
tomxor
1 Nickel

Re: XPS 13 9370: ath10k_pci firmware crash

For users of Debian based systems:

At the time of writing the firmware-atheros package in both Debian stable and testing appears to use a significantly older firmware than presently available in the ath10k-firmware repo.

On my system the hash for:

 

/lib/firmware/ath10k/QCA6174/hw3.0/firmware-6.bin
sha 3f57a1972dc15d25db5722e797e44ace1982d4a5

 

Matches the following version:

 

firmware-6.bin_WLAN.RM.4.4.1-00079-QCARMSWPZ-1
sha 3f57a1972dc15d25db5722e797e44ace1982d4a5

 

Whereas the latest version is:

 

firmware-6.bin_WLAN.RM.4.4.1-00128-QCARMSWPZ-1
sha db5a1865089d483ff3ab75d43726206afac562fa

 

 

The version provided by the apt repo is from Dec 2017, there have been 5 revisions since this one. I'm currently testing the latest above version, although I have no way of reproducing the issue reliably so I do not know if this has fixed my issue yet.

I replaced the board-2.bin and firmware-6.bin as per the killer's suggestions for manual installation of a previous revision of this same driver. i.e:

sudo su
cd /lib/firmware/ath10k/QCA6174/hw3.0/
git clone https://github.com/kvalo/ath10k-firmware.git

mv ./board-2.bin ./board-2.bin.bak
cp ./ath10k-firmware/QCA6174/hw3.0/board-2.bin ./board-2.bin

mv ./firmware-6.bin ./firmware-6.bin.bak
cp ./ath10k-firmware/QCA6174/hw3.0/4.4.1/firmware-6.bin_WLAN.RM.4.4.1-00128-QCARMSWPZ-1 ./firmware-6.bin

rm -rf ./ath10k-firmware
exit

 

... and then reboot.

 

0 Kudos
bkanuka
1 Copper

Re: XPS 13 9370: ath10k_pci firmware crash

@tomxor instructions are great but I'm still crashing with the `00128` firmware: 

 

[43555.888628] ath10k_pci 0000:02:00.0: firmware crashed! (guid 7d0a08f6-81a4-4616-9ff0-8b0bf703872d)
[43555.888649] ath10k_pci 0000:02:00.0: qca6174 hw3.2 target 0x05030000 chip_id 0x00340aff sub 1a56:143a
[43555.888655] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 1 tracing 1 dfs 0 testmode 0
[43555.890561] ath10k_pci 0000:02:00.0: firmware ver WLAN.RM.4.4.1-00128-QCARMSWPZ-1 api 6 features wowlan,ignore-otp,mfp crc32 4c523412
[43555.892065] ath10k_pci 0000:02:00.0: board_file api 2 bmi_id N/A crc32 4ed3569e
[43555.892074] ath10k_pci 0000:02:00.0: htt-ver 3.56 wmi-op 4 htt-op 3 cal otp max-sta 32 raw 0 hwcrypto 1
[43555.904017] ath10k_pci 0000:02:00.0: failed to get memcpy hi address for firmware address 4: -16
[43555.904019] ath10k_pci 0000:02:00.0: failed to read firmware dump area: -16
[43555.904021] ath10k_pci 0000:02:00.0: Copy Engine register dump:
[43555.904030] ath10k_pci 0000:02:00.0: [00]: 0x00034400   7   7   3   3
[43555.904039] ath10k_pci 0000:02:00.0: [01]: 0x00034800   5   4 177 178
[43555.904047] ath10k_pci 0000:02:00.0: [02]: 0x00034c00  48  47 110 111
[43555.904056] ath10k_pci 0000:02:00.0: [03]: 0x00035000   6   6   8   6
[43555.904064] ath10k_pci 0000:02:00.0: [04]: 0x00035400 6061 6043  14 206
[43555.904073] ath10k_pci 0000:02:00.0: [05]: 0x00035800   0   0  64   0
[43555.904081] ath10k_pci 0000:02:00.0: [06]: 0x00035c00   7   5   6   4
[43555.904089] ath10k_pci 0000:02:00.0: [07]: 0x00036000   0   1   0   1
[43555.930920] ath10k_pci 0000:02:00.0: failed to read hi_board_data address: -28
[43556.011455] ieee80211 phy0: Hardware restart was requested
[43556.738039] ath10k_pci 0000:02:00.0: Unknown eventid: 118809
[43556.741015] ath10k_pci 0000:02:00.0: Unknown eventid: 90118
[43556.836318] ath10k_pci 0000:02:00.0: device successfully recovered

This probably requires a fix beyond just a firmware upgrade...   

0 Kudos
ShedG
1 Nickel

Re: XPS 13 9370: ath10k_pci firmware crash

hbarta@rocinante:~$ grep "firmware crashed" dmesg.lst 
[24156.510250] ath10k_pci 0000:02:00.0: firmware crashed! (guid c0b1ed63-0be8-4b58-8560-31e0a56b4fde)
[48170.442566] ath10k_pci 0000:02:00.0: firmware crashed! (guid a46d0a8e-cbea-4a3f-8833-f5f85f7483ee)
hbarta@rocinante:~$

Perhaps the reason I have been losing WiFi connections recently.

I'm running Debian Stretch with 

hbarta@rocinante:~$ uname -r
4.19.0-1-amd64
hbarta@rocinante:~$ 

I'm using the firmware downloaded form the Killer Github repo. I thought I was having problems with Debian overwriting the firmware with updates so I wrote a script to compare installed firmware vs. what is available on Github.

 

0 Kudos