[SOLVED] Upgrade to Mageia 9 failed: "You are in emergency

[martin_ward]

Edit: Solution is to use UUID= for all mounted partitions in /etc/fstab

I have a machine with three SSD drives and four hard drives:

ASUS PRIME TRX40-PRO AMD Threadripper PCIe 4.0 ATX Motherboard
AMD Ryzen Threadripper 3970X 3.7GHz 32C/64T 128MB Cache, 280W CPU
Palit GeForce RTX2080 SUPER GameRock 8GB Graphics Card

/boot/EFI is mounted on a vfat partition nvme1n1p1 with Windows on nvme1n1p3

I have two root partitions (main and backup) on nvme0:

root1 is on nvme0n1p2 with UUID 1d91f854...
root2 is on nvme0n1p3 with UUID 10946a47...

(there are two other root partitions on nvme2)

These are copies of the same root partition, with custom /etc/fstab files, so I can boot from the menu into either root partition. I have a script to back up a root partition, so I can check if an update is OK and then back up root to another root partition.

Previously root1 was the first entry and root2 the second, and grub was reading /boot/grub2/grub.cfg on roo1 to provide the boot menu.

I booted into root2 to test the upgrade to Mageia 9. This seemed to go OK. When I rebooted I was surprised to see the first entry selected. I tried to boot from it and it failed: after the usual BIOS error messages
is says "You are in emergency mode. After logging in type "journalctl -xb" to view system logs,..."

I entered the root password to get a shell, rebooted and chose the second entry which booted into my old Mageia 8 root directory (which fortunately still seems to be working).

/boot/grub2/grub.cfg on root1 is unchanged, while there is a new file in /boot/grub2/grub.cfg on root2. The root2 file seems to be the one it is using for boot, since the second entry boots into roo1 while the first entry boots into root2 (and fails).

The file /boot/EFI/EFI/mageia/grubx64.efi has been updated.

I have attached a copy of the output of journalctl -xb, which I collected by booting into the root shell, a gzipped copy of this is attached.

Any help would be appreciated!

[doktor5000]

Hi there, would be helpful if you don't use journalctl -x as that adds a lot of clutter to the logs.
Would also be helpful if you would post the fstab from that system that you booted and least the output of lsblk or something like that.

Seems the system ends up in emergency mode, because it cannot mount all filesystems that would be mounted by local-fs.service (and which are not marked by either nofail or noauto).
At least swap, /boot/EFI and additionally all your NFS mounts (including /home) fail to mount. Also fsck.ext4 fails on /dev/nvme0n1p2

You should probably also rethink your hierarchical mounts, and at least add some failsafes using e.g. RequiresMountsFor= on the mount points that each mount unit requires for hierarchical mounts
See man page for systemd.unit for more details: https://www.freedesktop.org/software/sy ... .unit.html

Code: Select all

Dec 22 16:50:31 pingu7 swapon[1304]: swapon: /dev/nvme0n1p4: read swap header failed
Dec 22 16:50:31 pingu7 systemd[1]: dev-nvme0n1p4.swap: Swap process exited, code=exited, status=255/EXCEPTION
[...]
Dec 22 16:50:39 pingu7 mount[1584]: mount: /boot/EFI: wrong fs type, bad option, bad superblock on /dev/nvme1n1p1, missing codepage or helper program, or other error.
Dec 22 16:50:39 pingu7 mount[1584]:        dmesg(1) may have more information after failed mount system call.
Dec 22 16:50:39 pingu7 kernel: FAT-fs (nvme1n1p1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Dec 22 16:50:39 pingu7 kernel: FAT-fs (nvme1n1p1): bogus number of reserved sectors
Dec 22 16:50:39 pingu7 kernel: FAT-fs (nvme1n1p1): Can't find a valid FAT filesystem
[...]
Dec 22 16:50:39 pingu7 systemd-fsck[1604]: fsck.ext4: Bad magic number in super-block while trying to open /dev/nvme0n1p2
Dec 22 16:50:39 pingu7 systemd-fsck[1604]: /dev/nvme0n1p2:
Dec 22 16:50:39 pingu7 systemd-fsck[1604]: The superblock could not be read or does not describe a valid ext2/ext3/ext4
Dec 22 16:50:39 pingu7 systemd-fsck[1604]: filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
Dec 22 16:50:39 pingu7 systemd-fsck[1604]: filesystem (and not swap or ufs or something else), then the superblock
Dec 22 16:50:39 pingu7 systemd-fsck[1604]: is corrupt, and you might try running e2fsck with an alternate superblock:
Dec 22 16:50:39 pingu7 systemd-fsck[1604]:     e2fsck -b 8193 <device>
Dec 22 16:50:39 pingu7 systemd-fsck[1604]:  or
Dec 22 16:50:39 pingu7 systemd-fsck[1604]:     e2fsck -b 32768 <device>
Dec 22 16:50:39 pingu7 systemd-fsck[1605]: /dev/nvme2n1p2: clean, 862177/4096000 files, 5782463/16384000 blocks
Dec 22 16:50:39 pingu7 systemd-fsck[1607]: /dev/nvme2n1p3: clean, 862196/4096000 files, 5778217/16384000 blocks
Dec 22 16:50:39 pingu7 systemd[1]: Mounting mnt-server2.mount...
[...]
Dec 22 16:50:39 pingu7 mount[1672]: mount: /root1: wrong fs type, bad option, bad superblock on /dev/nvme0n1p2, missing codepage or helper program, or other error.
Dec 22 16:50:39 pingu7 mount[1672]:        dmesg(1) may have more information after failed mount system call.


[martin_ward]

Thank you very much for your reply!

I am not sure what you mean by "hierarchical mounts": the only mount that depends on another mount is nvme1n1p1 mounted on /boot/EFI which is on root.

All the NFS mounts are soft mounts: server2:/home is mounted on /mnt/server2 while the actual /home is on nvme1n1p5.

I have attached the fstab and the outputs of lsblk, lsblk -O, lsmod, swapon and journalctl -b from the failed root2 boot and also the successful root1 boot into Mageia 8.

As you can see, Mageia 8 happily mounts both swap partitions, /home, /boot/EFI, and (once the network is up) the nfs mounts.

I see that nouveau modules are running in lsmod, not the nvidia ones (only the Nvidia drivers will work for Mageia 8 at least).

fsck reports nvme0n1p2 is clean when booting on root1 (which is nvme0n1p2):

Dec 23 12:21:33 pingu7 systemd-fsck[1120]: /dev/nvme0n1p2: clean, 862099/4096000 files, 5788082/16384000 blocks

I hope this helps, and thanks again for looking into this.

[martin_ward]

I wonder if Mageia 9 is giving the partitions different numbers (different device names) than Mageia 8, so it is trying to mount the swap partition as ext4, or whatever?

When booting off root1 into Mageia 8, "lsblk -O" says that /dev/nvme0n1p2 is a 61.2G ext4 partition with UUID 1d91f854-5009-4055-bd55-19c325eb7533

When attempting to boot off root2 into Mageia 9, "lsblk -O" says that /dev/nvme0n1p2 is a "Microsoft reserved partition" with UUID d325d71b-94db-47b7-bfce-2c0a07ec2b16, while the 1d91f854... UUID is at /dev/nvme1n1p2

[martin_ward]

I edited /etc/fstab to change all the entries to use "UUID=...." to specify the partition, and now it is working!

Interestingly, after booting up all the partitions have the "expected" names: / is mounted on /dev/nvme0n1p2 and so on. So it seems that the initial boot process (initrd), before the "real" root partition is mounted, is getting different names, but by the time the system comes up, the names have changed.