[Solved] Mageia 5 Fails Normal Boot

[yankee495]

Hello,

I had a power failure that appears to have corrupted something. When I try to boot Mageia 5 it stops and keeps going back to "Started System Logging Service". It will continue booting in circles at that point.

When I boot to recovery mode and enter the root password, then just type exit, it will continue and fully boot.
That's right, just exiting without doing anything causes it to boot.

Also, booting to recovery mode and pressing Control D causes it to fully boot.

I'm assuming something is not being loaded in recovery mode that is loaded in normal mode and that's causing the problem.

My question is, what is not running in recovery mode that is running when booting normally? Are there normal and recovery mode logs I can look at?

Since the logging system has changed, I don't know which log is best to post.

Thank you.

[gohlip]

Assuming you have grub2 as bootloader (recovery mode? - should be)
At Mageia terminal, and as root,

Code: Select all

grub2-install /dev/sda
grub2-mkconfig -o /boot/grub2/grub.cfg

And to answer your question, failsafe boots without the plymouth and graphics kms stuff as well as not including swap.
If above still won't work, in sequence
1. take a look at your swap uuid in fstab compared with blkid
2. remove quiet and splash from your grub entry (not failsafe) and see if it boots - then it's your plymouth
3. remove quiet and splash and add '3' (no appos) and see if it boots to login prompt - then it's your graphic drivers - try 'startx' -
4. do dracut -f

[doktor5000]

Since the logging system has changed, I don't know which log is best to post.

Best attach the full log of

Code: Select all

journalctl -ab > /tmp/journal.log

It would also be sufficient if you do that when in recovery mode to find out what went wrong so that that system starts to recovery mode. Usually it's a filesystem check that needs to be done, or in your case the journal logs may need to be verified.

The latter can be done as root via

Code: Select all

journalctl --verify

See also https://www.freedesktop.org/software/sy ... l#--verify

[yankee495]

Thank you both, and hello Dok. I've been to a "people doctor" today and I'm just not at my best or I'd probably have this figured out.

I believe you posted just what I needed. I've done a lot of looking around but haven't changed anything and it's acting exactly as I described.
Booting to recovery and hitting Control D makes it boot to the login screen and then it goes into KDE.
My only problem is that it will not do a straight power on and boot. Once running, I don't see any problems and shutdown seems fine.

I hope this helps. I think I posted it correctly. I think the attached journal.log file shows the loop.

Here is the result of the journal log verification. I don't know how to repair the errors.

Code: Select all

[root@yankee john]# journalctl --verify
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000001e1eb1-00054718c4c8be61.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@0005512aef68b378-35e0762e7680a238.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@00055d8fca1a1ebf-a11798d646dccfd0.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000002dc5df-000547cb028d0cf6.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-0000000000092182-00054656c4f39077.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000000e6788-000546825039dfee.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@ded35a70c61d45cfa2ae438dc650648c-0000000000000001-00056184ef5d7f1a.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000544d18f9d8f7a-81e9dd3e6f41f909.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/user-1001.journal                                                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@51ca584b428e4ec68493cde94dd2a6d9-0000000000000001-000553f09b29e499.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system.journal                                                               
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000001b7b7a-000547038bbb095d.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/user-500.journal                                                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000002b2a65-000547bf3b6b7b74.journal
1053fb0: invalid entry item (1/20 offset: 000000                                                                                     
1053fb0: invalid object contents: Bad message                                                                                         
File corruption detected at /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000545583db25380-10591db8e8b3db4c.journal~:1053fb0 (of 25165824 bytes, 68%).
FAIL: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000545583db25380-10591db8e8b3db4c.journal~ (Bad message)
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-0000000000386a50-000543d924486c51.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@3b7c91c500ef4e45ad814a98ee16cafb-000000000001ed3d-00054c81fef906d8.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000000bc551-000546695431aa9e.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@0005591e6bce9375-ee69409e6525ec85.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000563f93e8c58a4-00deb533f72661b2.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-00000000002d6f14-000542797772b8b5.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-0000000000234fac-0005475dfb008aca.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-0000000000163d35-000546cc52a7af36.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-00000000002ad0f3-000542577fb5a92d.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-0000000000350577-000543d17a8566bd.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/user-61000@beefa196db0b420bb35f7fabd6fdaba8-00000000000172d6-000545879dc33fef.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@51ca584b428e4ec68493cde94dd2a6d9-000000000000f89c-00055655064a237f.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@02122f2d16fc451d9a3a0e63cfcae4d4-0000000000000001-00055d8fca188990.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@00054a178b6f26de-9b4a20e4898a7bcb.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-0000000000000001-000545583db11eed.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000000172e5-000545879dc344e6.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@0005496633255255-649025faebee011f.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@00056184ef5f1c1e-7fca60fba494039b.journal~                                                                   
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-000000000025de68-000542375b439559.journal                                   
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@b45562f56c0c474aa605b042d580396c-0000000000000001-0005591e6bccaaf2.journal                                   
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-0000000000069888-000545f22c172e22.journal                                   
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-000000000032f8f9-0005483380794a6b.journal                                   
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-00000000002859ce-0005424308188e33.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000000403dd-000545b4be79ef33.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000552e93212dd2f-0168c0fe7a0ef2b0.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-0000000000323eea-000542c6801b15a5.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@16153fe219d34d88acd5292ed06fe753-0000000000000001-00054e21eb43394a.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/user-61000.journal                                                           
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000003aa684-0005490edcde5677.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000003587ff-00054862a7d4bce4.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-0000000000288725-000547ab5bc7fe20.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000563f24a790295-402607181819d413.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/user-1005.journal                                                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000553f09b2a8d32-107b75fa760645b4.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-00000000003005db-000542938434da8c.journal
310efd8: unused data (entry_offset==0)                                                                                               
310efd8: invalid hash (7b4c0206e44fa861 vs. 9313cfcd0e74f52e                                                                         
310efd8: invalid object contents: Bad message                                                                                         
File corruption detected at /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@00054e21eb43fe74-e6c9938697d7195a.journal~:310efd8 (of 58720256 bytes, 87%).
FAIL: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@00054e21eb43fe74-e6c9938697d7195a.journal~ (Bad message)
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000543dc9947130b-2a5102fb8a8f53d4.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000003055a3-0005481edfe203d6.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-00000000003be251-000543daba2f9395.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-000000000018e0d6-000546e90428a69d.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-0000000000139ae9-000546b875e1e3a5.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-000000000020b885-000547372c534bcd.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@07b45d86ea8a4fedaac197951aead8f9-00000000002347f5-0005422354393d74.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/user-488.journal                                                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@3b7c91c500ef4e45ad814a98ee16cafb-0000000000000001-00054a178b6dd924.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-0000000000381c07-000548ccd9867099.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@000563f889c4356e-8bc16dc8f9ab0aeb.journal~                             
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-000000000025eb13-00054775f90e8cc8.journal
PASS: /var/log/journal/0edc163ab40143e2bb1c9edf1f273561/system@388049e13c6e4a1d9350d0a373cd6747-00000000001100b6-000546a4d55e0ea1.journal
[root@yankee john]#

And here is journal.log

[yankee495]

I installed Mageia 6 on a different drive. By the way, I love it!

The short story is that after looking all over it was as simple as running fsck on the /home partition while it was unmounted.
I booted to Mageia 6 and ran fsck. it would not allow the -p option so I used -y and a few seconds later it was fixed.

I appreciate the assistance and now I'm off to study my chances of successfully upgrading Mageia 5 to 6.
It sure would save me a lot of work.

This is the command I used as root. Change the device to the one you want to repair. It must be unmounted.

Code: Select all

fsck -y -v /dev/sdb6

[doktor5000]

Here is the result of the journal log verification. I don't know how to repair the errors.

Well, the bad thing is, you can't repair them or drop the corrupted bits (usually those ending with a ~). Upstream does not see a usecase for that, read the reply to my question about that: https://bugs.freedesktop.org/show_bug.cgi?id=64116

[yankee495]

Sorry Dok,

I have read that exact bug report. I should've posted that I'd found the solution to that, or lack of one.
The way I understand it is I shouldn't delete them and there isn't a way to make them "PASS" again.

So, in your opinion, what's the best file system to use? Should I be using a more robust file system with better recovery capabilities or does one even exist? I know that's a loaded question, which his best always seems to get people fired up expressing their opinion why you're wrong. I really should get a UPS because we do have power failures regularly. Not that we have them a lot, but you can count on a few per year, and if you're lucky you may only be using the machine a couple of times when it hits. I used to have a good UPS but it burned up in a house fire and I just never replaced it. Oh, it was not the cause of the fire.

I have done a lot of customization to my Mageia install, from a custom short date format to a few tricks to get my favorite screensaver to run. I compile some of my programs to get the newest version, such as Darktable. I've set up a RAM disk with no swap and tons of things. My point is, it'd take a lot of work to get it back where it is and I should take steps to protect it a little better. I think that's probably true for most people and a basic UPS isn't as expensive as they used to be.

Oh, I think I knew who you were and forgot. You're almost famous! I'm pretty good at not breaking my system, and fixing things, so when something happens that I can't fix, I sure appreciate your help. Thanks a bunch.

[jiml8]

You did not ask me, but I will tell you anyway.

Irrespective of UPS or reliable power, or this filesystem vs that filesystem, or anything else, if you wish to retain your data and your configuration regardless of what happens. you should back up your system.

This isn't hard to do and the effort spent today will without question save you down the road.

My backup system has allowed me to recover from double hard drive failures (two drives failed, with the system on one and one backup of the system on the other one), bad upgrades, corruption from power failures, corruption from I-don't-know-what, and just fatfingering and deleting critical stuff by mistake.

No matter what I do to it or what happens to it, I can usually recover my system in no more time than it takes to rsync last night's backup into place. It takes longer, of course, if recovery involves replacing hardware.

[doktor5000]

The way I understand it is I shouldn't delete them and there isn't a way to make them "PASS" again.

Yes, you cannot make them "PASS" again, but you can delete all those ending with ~ as those are corrupted.
If you do not need older journal files you can also try via

Code: Select all

journalctl -–vacuum-size=1M

which will remove archived journal files above 1MB overall size, so essentially all of your journal.

Then run --verify again to see if there are still some corrupted files, you can delete those, then restart systemd-journald to rotate any logs that are active.

You can also try to disable compression as mentioned in http://centosfaq.org/centos/systemd-jou ... orruption/ if you still frequently see corrupted files, if your journal is heavily frequented.

[yankee495]

You did not ask me, but I will tell you anyway.

Irrespective of UPS or reliable power, or this filesystem vs that filesystem, or anything else, if you wish to retain your data and your configuration regardless of what happens. you should back up your system.

No matter what I do to it or what happens to it, I can usually recover my system in no more time than it takes to rsync last night's backup into place. It takes longer, of course, if recovery involves replacing hardware.

I bought a 3TB HDD with this exact thing in mind but haven't taken the time to set it up. I was planing to do it after upgrading to Magiea 6 and after learning how to do it. I've never done anything besides a plain system image. You know, the various programs you can boot to on a DVD and copy an ISO of your drive back to your drive. I realize Mageia has much better and easier (in the long run) options to do this but have never tried them.

I do have the space and today I'm checking the drives for errors and looking into it. I don't want to back up errors that need fixed and have to fix them again after a restore. I know I didn't ask but thanks for the info. I was getting around to it if I needed help.

Thanks for all of the info Dok. I'll give it a good look and see what I have going on. I understood that I couldn't make them pass again but wasn't clear on what I could or could not delete. I have plenty of space but don't want stuff just lying around if it's not needed.

A very strange thing just happened. I couldn't launch ksnapshot from a icon on the panel and when I hit the Kicker icon the whole thing crashed. The panel disappeared and things didn't look good. I have my power button set up for shutdown when pressed, so I pressed it and it hung on shutdown. It booted back up fine, but I need to look around. I may have a loose or bad cable or SSD going bad. Of course, it could be the errors on the drive, I'm not sure at this point. It's a very rare thing for the whole thing to crash. So rare that I don't remember the last time it happened.

Thanks again guys and I'm ok unless I need a little help with the backup process. I'll look around and see if I can figure it all out, I should be able to get it.

[doktor5000]

A very strange thing just happened. I couldn't launch ksnapshot from a icon on the panel and when I hit the Kicker icon the whole thing crashed. The panel disappeared and things didn't look good. I have my power button set up for shutdown when pressed, so I pressed it and it hung on shutdown. It booted back up fine, but I need to look around. I may have a loose or bad cable or SSD going bad. Of course, it could be the errors on the drive, I'm not sure at this point.

In that case I'd have switched to a tty and checked the remaining processes and the journal. Apart from that, I have my box set up to forward all messages to tty12 as it was the case earlier on nearly all distros where you had either tty8 or tty12 which ran dmesg.
See https://wiki.archlinux.org/index.php/Sy ... ev.2Ftty12
If that doesn't work anymore, you can still use the Magic SysRq keys to at least sync and safely unmount the remaining partitions.
See e.g. https://en.wikipedia.org/wiki/Magic_SysRq_key

[yankee495]

Thanks DOk, that's great info, didn't know about dmesg in the old days. I'll take all of the help I can get from the system because I'm certain it knows more than I do. I did install a drive into the case and I guess it's possible I knocked a cable a bit loose, but I couldn't tell by looking. I did double check everything and it's running great now but I am going check out what you posted. In fact ,I'm going to print that Magic info, looks very useful.

[doktor5000]

In fact ,I'm going to print that Magic info, looks very useful.

No need to print it out the whole page, remember trees need to die for that

You should remember REISUB, which is often referred to as "Reboot Even If System Utterly Broken" or "Raising Elephants Is So Utterly Boring" or BUSIER backwards. https://en.wikipedia.org/wiki/Magic_SysRq_key#Uses