kernel 4.9.50 with NVMExpress M.2 SSD

[schnuck]

Hello Forum,

All new kernels cannot boot. They stop while mounting the root device, which is a NVMExpress SSD.
It is called /dev/nvme0np5 by the running kernel 4.9.40.

I guess the new kernels need the matching module for that, or it seems the automatic dkms support
for this is not working. So I have to force the installation or compilation of the right kernel module.

Any hints on this?

Greetings

[tmb]

Oh they do work on NVME, I have an AMD Ryzen basted workstation at home that happily boots/runs 4.9.50, so we need more info...

And nvme support is in upstream kernel, so no dkms involved in that part... or are you thinking of nvidia drivers ?

[schnuck]

Oh they do work on NVME, I have an AMD Ryzen basted workstation at home that happily boots/runs 4.9.50, so we need more info...

And nvme support is in upstream kernel, so no dkms involved in that part... or are you thinking of nvidia drivers ?

The kernel has a panic while booting and it says "cannot mount root fs on unknown block device (0,0)" (or something very similar to this).
So i assume it's not recognizing nvme devices.

Also I noticed that there was no initrd-4.9.50-desktop-1.mga6.img created by installing this kernel.
I created it by hand - but probably it's missing nvme?

Greetings

[schnuck]

Also: I cannot find "nvme" starting with /etc/ in any file - apart from fstab and default/grub.
There must be some info that the kernel needs nvme before mounting the root fs - or not?

[doktor5000]

There must be some info that the kernel needs nvme before mounting the root fs - or not?

Nope, the same way as it doesn't need to know whether a disk is connected via SCSI, SATA, PATA or maybe USB or something like that.

[gohlip]

Just adding my 2 cents.

Does '# parted -l' or '# blkid' or .... not show nmve partitions?
And do you not set /boot/efi to its fat32 partition, say /dev/nvme0np1 ?
If yes, and it still fails to boot, at install media, install packages efivar, efibootmgr, grub2-common, grub2-efi, dosfstools and reinstall grub (2) setting up mount points to /dev/nvme0np5 and /dev/nvme0np1.

[schnuck]

Just to point it out clear.
I am running kernel 4.9.40 and it works. I can boot this kernel without any problems.
But upgrading to 4.9.43 did not work, and upgrading to 4.9.50 did also not work.
With either of the later kernel it ends with a kernel panic which says "unknown block (0,0)"

Greetings

[wintpe]

as the first reply stated we need more info.

so start with what motherboard, processor and nvme model, because if its working for some there has to be a subtle difference in yours. also check if there are any firmware updates that list issues with nvme support.
for your drive and motherboard

regards peter

[schnuck]

Hello everybody,

Sorry for not answering for a while - I just went on using the older kernel, kind of accepting the problem.

But now I wanted to upgrade to kernel 4.9.56 and so I did that with rpmdrake. I installed the necessary dependancies and all kernel modules and video drivers and the like and it wasn't working -- same problem as with every kernel I tried to upgrade to.

I digged further into the problem and found the following solution:

I create an initial ram disk by hand (with the nvme driver added to it):

mkinitrd initrd-4.9.56-desktop-1.mga6.img 4.9.56-desktop-1.mga6 --preload --force

After that I update grub2 (by running update-grub2) and voila it works!

At least I have a working solution now. Now I can boot into 4.9.50 and 4.9.56 and presumably all newer kernels.

Greetings

[schnuck]

Sorry, of course I run it with nvme added to it:

mkinitrd initrd-4.9.56-desktop-1.mga6.img 4.9.56-desktop-1.mga6 --preload nvme --force

[wintpe]

I have an NVME ssd on my new hardware, and thats having no issues with any of those kernels.

the nvme module is detected and loaded at install and subsequently no problem.

I cant see why building mkinitrd has helped.

and so far youve not helped us to help you with anything but discussion.

Discussion wont help nail down a problem.

we need an lspci output (not all of it, just for your device)
to find it, run dmesg|grep nvme

then look for the line that contains some of the numbers in that dmesg ie

00000:9:0.0

and it should say samsung or whatever manufacturer nvme express device.

also perhaps this is fixed in the backports testing kernels worth a try.

regards peter

[schnuck]

Hello Peter, Hello Forum,

I'm sorry guys, for a while I just ignored the problem, because there was no obvious reason to use the latest kernel.
A few days ago I just tested it again and found a working solution or workaround it.

But if the problem is worth being understood I will post some infos about it.

Here is output of dmesg|grep nvme:

Code: Select all

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.56-desktop-1.mga6 root=UUID=396ef858-45b1-4949-992a-7b42e61d51a8 ro splash quiet nokmsboot resume=/dev/nvme0n1p2 audit=0 acpi_backlight=vendor net.ifnames=0 biosdevname
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.56-desktop-1.mga6 root=UUID=396ef858-45b1-4949-992a-7b42e61d51a8 ro splash quiet nokmsboot resume=/dev/nvme0n1p2 audit=0 acpi_backlight=vendor net.ifnames=0 biosdevname
[    1.226169] PM: Checking hibernation image partition /dev/nvme0n1p2
[    2.236250] nvme nvme0: pci function 0000:3d:00.0
[    2.349031]  nvme0n1: p1 < p5 p6 p7 p8 > p2
[    2.807794] EXT4-fs (nvme0n1p5): mounted filesystem with ordered data mode. Opts: data=ordered
[    2.850281] EXT4-fs (nvme0n1p5): mounted filesystem with ordered data mode. Opts: data=ordered
[    2.864187] dracut: Mounted root filesystem /dev/nvme0n1p5
[    2.893457] dracut: Checking ext4: /dev/nvme0n1p7
[    2.893798] dracut: issuing e2fsck -a  /dev/nvme0n1p7
[    2.904345] dracut: /dev/nvme0n1p7: clean, 257132/639584 files, 1535895/2558091 blocks
[    2.909367] EXT4-fs (nvme0n1p7): mounted filesystem with ordered data mode. Opts: acl
[    3.097038] systemd[1]: usr.mount: Unit is bound to inactive unit dev-nvme0n1p7.device. Stopping, too.
[    3.118555] EXT4-fs (nvme0n1p5): re-mounted. Opts: acl
[    3.120302] EXT4-fs (nvme0n1p7): re-mounted. Opts: acl
[    3.265626] Adding 625960k swap on /dev/nvme0n1p2.  Priority:-1 extents:1 across:625960k SSFS
[    3.297582] EXT4-fs (nvme0n1p8): mounted filesystem with ordered data mode. Opts: acl
[    3.301358] EXT4-fs (nvme0n1p6): mounted filesystem with ordered data mode. Opts: acl
[    3.415829] EXT4-fs (nvme0n1p5): re-mounted. Opts: acl,commit=0
[    3.417089] EXT4-fs (nvme0n1p7): re-mounted. Opts: acl,commit=0
[    3.420783] EXT4-fs (nvme0n1p8): re-mounted. Opts: acl,commit=0
[    3.421725] EXT4-fs (nvme0n1p6): re-mounted. Opts: acl,commit=0



Greetings

[wintpe]

so the second part of that was lspci|grep 3d:00
regards peter

[schnuck]

Sorry, forgot that second command. "lspci | grep 3d:00" :

Code: Select all

3d:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)


[wintpe]

OK, not a diagnosis, but ill update this later with what mine is, the first part is the same, im just not sure if the last bit SM951/PM951 is the same on mine from memory.
so when i get home tonight ill update this.

it could be that the kernel is not detecting and loading that device in the shipped nvme module.

also look at the samsung site, they do firmware updates for drives, you basically boot an iso and then a mini linux starts and tells you if your drive needs updating.

be wary of the possibility that firmware updates can wipe or toast your drive.

can you run modinfo nvme

regards peter

[schnuck]

Output of modinfo nvme:

Code: Select all

filename:       /lib/modules/4.9.56-desktop-1.mga6/kernel/drivers/nvme/host/nvme.ko.xz
version:        1.0
license:        GPL
author:         Matthew Wilcox <willy@linux.intel.com>
srcversion:     1A6F7254BEC3E9B31BED2BE
alias:          pci:v0000106Bd00002003sv*sd*bc*sc*i*
alias:          pci:v0000106Bd00002001sv*sd*bc*sc*i*
alias:          pci:v*d*sv*sd*bc01sc08i02*
alias:          pci:v00001C5Fd00000540sv*sd*bc*sc*i*
alias:          pci:v00001C58d00000003sv*sd*bc*sc*i*
alias:          pci:v00008086d00005845sv*sd*bc*sc*i*
alias:          pci:v00008086d00000A54sv*sd*bc*sc*i*
alias:          pci:v00008086d00000A53sv*sd*bc*sc*i*
alias:          pci:v00008086d00000953sv*sd*bc*sc*i*
depends:        nvme-core
intree:         Y
vermagic:       4.9.56-desktop-1.mga6 SMP mod_unload
parm:           use_threaded_interrupts:int
parm:           use_cmb_sqes:use controller's memory buffer for I/O SQes (bool)


[wintpe]

ok heres mine

Code: Select all

09:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961

filename:       /lib/modules/4.9.56-desktop-1.mga6/kernel/drivers/nvme/host/nvme.ko.xz
version:        1.0
license:        GPL
author:         Matthew Wilcox <willy@linux.intel.com>
srcversion:     1A6F7254BEC3E9B31BED2BE
alias:          pci:v0000106Bd00002003sv*sd*bc*sc*i*
alias:          pci:v0000106Bd00002001sv*sd*bc*sc*i*
alias:          pci:v*d*sv*sd*bc01sc08i02*
alias:          pci:v00001C5Fd00000540sv*sd*bc*sc*i*
alias:          pci:v00001C58d00000003sv*sd*bc*sc*i*
alias:          pci:v00008086d00005845sv*sd*bc*sc*i*
alias:          pci:v00008086d00000A54sv*sd*bc*sc*i*
alias:          pci:v00008086d00000A53sv*sd*bc*sc*i*
alias:          pci:v00008086d00000953sv*sd*bc*sc*i*
depends:        nvme-core
intree:         Y
vermagic:       4.9.56-desktop-1.mga6 SMP mod_unload
parm:           use_threaded_interrupts:int
parm:           use_cmb_sqes:use controller's memory buffer for I/O SQes (bool)

[root@localhost ~]# df -k
Filesystem     1K-blocks     Used Available Use% Mounted on
devtmpfs        16400784        0  16400784   0% /dev
tmpfs           16439348        0  16439348   0% /dev/shm
tmpfs           16439348     1452  16437896   1% /run
/dev/nvme0n1p2  51643864  7082112  41908620  15% /
tmpfs           16439348        0  16439348   0% /sys/fs/cgroup
tmpfs           16439348       20  16439328   1% /tmp
/dev/nvme0n1p1    305576      136    305440   1% /boot/EFI

[root@localhost ~]# dmesg|grep nvme
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.56-desktop-1.mga6 root=UUID=436f4304-6782-429e-9598-7c4b930d693c ro splash quiet noiswmd nokmsboot resume=/dev/nvme0n1p3 audit=0
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.56-desktop-1.mga6 root=UUID=436f4304-6782-429e-9598-7c4b930d693c ro splash quiet noiswmd nokmsboot resume=/dev/nvme0n1p3 audit=0
[    1.268952] PM: Checking hibernation image partition /dev/nvme0n1p3
[   65.043757] nvme nvme0: pci function 0000:09:00.0
[   65.261688]  nvme0n1: p1 p2 p3 p4
[   65.281137] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: data=ordered
[   65.314622] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: data=ordered
[   65.329430] dracut: Mounted root filesystem /dev/nvme0n1p2
[   65.527609] EXT4-fs (nvme0n1p2): re-mounted. Opts: acl
[   65.647075] Adding 4193292k swap on /dev/nvme0n1p3.  Priority:-1 extents:1 across:4193292k SSFS
[   65.653265] FAT-fs (nvme0n1p1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
[   65.671608] EXT4-fs (nvme0n1p4): mounted filesystem with ordered data mode. Opts: acl




and just to re-iterate ive not had to do anything special to get that module to load.

have we already discussed system bios, and ssd bios might be an option, have you tried clearing to defaults your bios setup

happy to post any more comparisons i can, I know from your point of view it looks like a software issue, but then when you see a system
with similar components not seeing the same issue, you have to be open minded about where the problem actually lies.
and try other things.

regards peter

[wintpe]

one thing ive not asked, what is the model of the drive, mines samsung 960 evo

you say nvme express, but what model drive.

regards peter