[CLOSED][WA]Kernel 5.10.x SSD drive performance regression

[Palin]

Hi there,
I had a huge performance hit on the SATA SSD I have on my Dell Latitude 3590 when I updated to the new kernel-desktop-5.10, both .6 and .8 (I'm just not sure about the .6 I didn't make lots of test). I switched back to 5.7.19-3.

What happened is that I/O is visually terribly slow, for instance I have a Terminator profile that usually takes less than a second to open six terminals with their bash, with 5.10 it takes more than 10 seconds from click to the bash prompt. Next case is Libreoffice, take Calc for example: it currently takes 2-3 seconds after boot (so nothing is in the memory cache), with 5.10 takes several seconds up to 30.

I use the laptop on my job so I'm unable to do many tests, but I can report configurations, settings, and so on if needed.

My opinion is that 5.10 introduced some defaults with does not play well with SATA SSD... I don't think filesystem is the culprit, it mostly ext4 by the way except EFI.

Maybe some of you into kernels can point my to a bug report or a workaround...

[magic]

Is there anything noteworthy in the log with running the new kernel?

Does it feel like the drive locks up randomly (maybe becoming more normal after several minutes) or is it just consistently slow?

[doktor5000]

Nope neither in dmesg nor in messages and all syslog files (I enabled rsyslogd). My feeling is that the bandwith is lower, the graph in the gnome system monitor extension shows a steady flow of 7 MiB/s (with spikes to 10) while with the rollbacked kernel is more than 100MiB/s (I just tried with a dd from /dev/zero, it goes up to 250 while write behind it's in effect, then goes down to ~100-120MiB/s), so I'm pretty sure it's not a spike. Of course it's mitigated when memory cache gets populated in normal use.

It's pretty a huge performance hit, it seems like having a magnetic drive (yet a fast one).

[tis]

Hi,
I think I cannot help, but I'm using this kernel with sata hdd's and nvme m.2 ssd and I haven't experienced any difference. I rolled back and tested sdd with Disks (gnome-disk-utils). The result is the same with 5.10 and prev. kernel. Your SSD's bad result is near my HDD's...
Your laptop screwed with different type of disks and I think it can be useful if you can specify your hardware (chipset,mem,cpu), maybe that will help...

Code: Select all

lspci -v

[doktor5000]

while with the rollbacked kernel is more than 100MiB/s (I just tried with a dd from /dev/zero, it goes up to 250 while write behind it's in effect, then goes down to ~100-120MiB/s), so I'm pretty sure it's not a spike.

You may want to elaborate on the /dev/zero part and how you actually tested.

[Palin]

Your laptop screwed with different type of disks and I think it can be useful if you can specify your hardware (chipset,mem,cpu), maybe that will help...

This is some information both from lspci and dmidecode, this is data from the “working” kernel:

Code: Select all

# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 08)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 620 (rev 07)
00:04.0 Signal processing controller: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem (rev 08)
00:14.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-LP Thermal subsystem (rev 21)
00:15.0 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 (rev 21)
00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI #1 (rev 21)
00:17.0 SATA controller: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] (rev 21)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #5 (rev f1)
00:1c.5 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point LPC Controller/eSPI Controller (rev 21)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-LP PMC (rev 21)
00:1f.3 Audio device: Intel Corporation Sunrise Point-LP HD Audio (rev 21)
00:1f.4 SMBus: Intel Corporation Sunrise Point-LP SMBus (rev 21)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
02:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78)


Code: Select all

# dmesg |grep -i sd|ata
[...]
[    0.643250] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 2 ports 6 Gbps 0x1 impl SATA mode
[    0.643273] ahci 0000:00:17.0: flags: 64bit ncq stag pm led clo only pio slum part deso sadm sds apst
[    0.643741] ata1: SATA max UDMA/133 abar m2048@0xdf233000 port 0xdf233100 irq 124
[    0.950775] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    0.964675] ata1.00: ATA-10: Crucial_CT525MX300SSD1,  M0CR031, max UDMA/133
[...]
[    0.643273] ahci 0000:00:17.0: flags: 64bit ncq stag pm led clo only pio slum part deso sadm sds apst
[    0.992237] sd 0:0:0:0: [sda] 1025610768 512-byte logical blocks: (525 GB/489 GiB)
[    0.994403] sd 0:0:0:0: [sda] Write Protect is off
[    0.995494] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    0.995500] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.997310]  sda: sda1 sda2 < sda5 sda6 sda7 >
[    0.998696] sd 0:0:0:0: [sda] supports TCG Opal
[    0.999219] sd 0:0:0:0: [sda] Attached SCSI disk
[    3.096592] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: data=ordered
[    3.198811] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: data=ordered
[    3.274982] dracut: Mounted root filesystem /dev/sda1
[    4.357895] EXT4-fs (sda1): re-mounted. Opts: acl
[    4.359880] sdhci: Secure Digital Host Controller Interface driver
[    4.360816] sdhci: Copyright(c) Pierre Ossman
[    4.925535] Adding 7332404k swap on /dev/sda5.  Priority:-2 extents:1 across:7332404k SSFS
[    7.526823] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: acl



CPU (from lscpu): Model name: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz

RAM is 16GB in two SODIMM banks (serials deleted), here's a single bank:

Code: Select all

        Array Handle: 0x0041
        Error Information Handle: Not Provided
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 8192 MB
        Form Factor: SODIMM
        Set: None
        Locator: DIMM B
        Bank Locator: Not Specified
        Type: DDR4
        Type Detail: Synchronous Unbuffered (Unregistered)
        Speed: 2400 MT/s
        Manufacturer: 80AD000080AD
        Serial Number: xxxxx
        Asset Tag: 01184900
        Part Number: HMA81GS6AFR8N-UH   
        Rank: 1
        Configured Memory Speed: 2400 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V



All SMART data on the SSD are pretty OK.

Note: the notebook has an m2 slot but I'm not using it since I had a pre-installed system on my SATA SSD which I moved from another system a while ago (I don't think the drive is the problem here).

[Palin]

You may want to elaborate on the /dev/zero part and how you actually tested.

It's a simple:

Code: Select all

# dd if=/dev/zero of=file_to_ssd_disk bs=1G count=10


to write zeroes on a 10GB file to an ext4. It's the simplest way to reproduce the “slowness”, which occours with every other piece of sofrware reading or writing from disk.

[tis]

Hi, TY,

It was interesting for me, so I've retried it with 1000MB data 10 times. The result is different as the data is more so the cache can't really effect, but there is no significant differences between kernel 5.10 and 5.7 at my machine. ( the result: read speed ~2GB/s; write speed is more interesting started from 450MB/s and after the 2nd and 3rd tests it is falls and after the 10th is is near 150MB/s but it is very similar in diff. kernels )

As I said I use sata hardisks, they are in raid mirror, so I could test it as an mdadm raid device:

Test result 1000MB 10x with SATA HDD RAID

Screenshot_20210129_Raid.png (119.32 KiB) Viewed 7027 times

So I can't see any differences, as I'm thinking our hardware is different:
- my computer is desktop - yours is laptop
- my chipset/sata/nvme controller is different
- my cpu is a bit older and different
So I think you should find hardware specific kernel differences (kernel driver module,kernel settings for laptop, mobile cpu,...) I hope I could help a little....

My computer:

Code: Select all

lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       39 bits physical, 48 bits virtual
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  2
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               158
Model name:          Intel(R) Core(TM) i3-7100 CPU @ 3.90GHz
Stepping:            9
CPU MHz:             3900.382
CPU max MHz:         3900,0000
CPU min MHz:         800,0000
BogoMIPS:            7799.87
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            3072K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d

lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 06)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 630 (rev 04)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:14.0 USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
00:17.0 RAID bus controller: Intel Corporation SATA Controller [RAID mode]
00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)
00:1b.2 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #19 (rev f0)
00:1b.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0)
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
00:1c.2 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #3 (rev f0)
00:1c.3 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #4 (rev f0)
00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0)
00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation 200 Series PCH LPC Controller (Z270)
00:1f.2 Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
00:1f.4 SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
02:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
05:00.0 Ethernet controller: Qualcomm Atheros Killer E2500 Gigabit Ethernet Controller (rev 10)
08:00.0 Non-Volatile memory controller: Intel Corporation SSD Pro 7600p/760p/E 6100p Series (rev 03)


[tis]

PS. I haven't deal with it yet, but Now I realize that my nvme ssd write speed is less than a half of Intel specified as max., but read speed can reach near 3GB/s. I think maybe there are some 'sw cache' thing for "Doors10" maybe I should find some Linux solution.
----
I have to edit: Intel has something called Intelmas CLI tools for Linux, it can manage nvme. I haven't read everything about it, there are several options, firmware update opportunity and something about performance, maybe this is I'm searching.

[Palin]

The controller seems to be this:

Code: Select all

# lspcidrake |grep -i -e sata -e sda
unknown         : Intel Corporation|Sunrise Point-LP SATA Controller [AHCI mode] [STORAGE_SATA] (rev: 21)


As you can see even mageia doesn't know which module is, I suppose it's supported directly into the kernel.

This is the equivalent dmesg entry:

Code: Select all

[    0.643250] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 2 ports 6 Gbps 0x1 impl SATA mode


It may have nothing to do with the driver per se but it may be that some default changed and I have to tune something in the kernel, I was watching even cpufreq and something like this by I have literally no clues.

[magic]

What is the APM level for your drive (both on the old and the new kernel)? Just been reading that the MX300 has been known to aggressively degrade its performance for power saving.

[Palin]

On the old (current) kernel it gsmartmontools reports: 254 (maximum performance)

I'll have a look as soon as I can reboot the machine on the 5.10.x

[Palin]

It seems that you were right, on new kernel it reports: 1 (minimum power consumption with standby)

How can I change that?

[doktor5000]

You can query and set it with hdparm.
Have a look at e.g. https://wiki.archlinux.org/index.php/Hd ... _udev_rule
Would still be interesting to know why the defaults have changed.

[Palin]

Thank you very much I tried the hdparm and it works, I tried the /etc/sysconfig/harddisks in the EXTRA_PARAMS= but it does not survive reboots.
At the moment I put hdparm into rc.local, I'll try udev next time I'll use that device.

[tis]

I'm happy that you could find the problem.

+ I realized that maybe I have some other problem. The Intel 7600p read speed could reach 3GB/s and the write speed could reach 1.3GB/s, I haven't reached at my examples. I've made some more test and when I've changed the data size from 10MB to 200MB i could reach the 3.1GB/s in Disks test util but the write speed is maximum was only 450MB/s.

I've mentioned Intel MAS tool, there is nothing to setup about it, and most of the options is not supported by my device.

I've searched the net about my ssd write speed problem and I found that maybe the test prog/way is bad:

Code: Select all

dd if=/dev/zero of=tempfile bs=10M count=100

can write with near 640MB/s

But

Code: Select all

dd if=/dev/zero of=tempfile bs=10M count=100 oflag=direct

can write with near 1.3GB/s

So there is no problem, just the test were wrong....

[doktor5000]

+ I realized that maybe I have some other problem.

Please don't hijack other peoples threads, create a separate one yourself if you have an issue.

[Palin]

I will mark the thread as SOLVED when I'll reach a point with udev where the boot itself it's back to being fast, rc.local of course applies too late in the boot sequence.

[tis]

Hi,

1, at Disks (gnome-disk-utility) you can manage on GUI
or
2, and I found you can create etc/hdparm.conf, something like

Code: Select all

/dev/sda {
apm = 254
apm_battery = 254
}


edited: -B removed seen in hdparm source example config file

[Palin]

I currently can't confirm hdparm in udev works:

Code: Select all

daemons/warnings.log.1:Feb  1 08:09:17 gmerigo-nb64 systemd-udevd[571]: Process '/usr/bin/hdparm -B 254 -S 0 /dev/sda' failed with exit code 1.

I tested and the -B parameters stays to 1. It would be nice to know why or how this default has been changed in 5.10.x kernels (I received an update today and it has the same behaviour).

I tried removing -S 0 and next reboot we will see if it works.

[Palin]

Found the culprit, my mistake, in mageia hdparm in in /usr/sbin/hdparm, this is way the udev rule in the archlinux link did not work.

Setting the thread to [CLOSED][WORKAROUND] as in my opinion the behaviour should not change.