[SOLVED] After update, can only boot in safe mode

[djohnston]

I'm still running Mageia3-beta1-i586 in VirtualBox. It has been updated regularly all this time. Except for one update hiccup, it has worked flawlessly, until now. Now I can only boot in safe mode. After having logged in, I started an update by opening the control center and selecting the software updater. There were two packages updated. I believe they were glibc and glibc-dev. Then I received a message that the system should be rebooted. I clicked the OK button, and another list of updates showed. I did not apply the second set of updates. I rebooted.

After logging in again, I started an update by opening the control center and selecting the software updater. There were a lot of updates, including a kernel upgrade from 3.8.5-desktop-1.mga3 to 3.8.6-desktop=2.mga3. After applying the updates, I once again received a message that the system should be rebooted. After clicking the OK button, I rebooted. That was the last time I could boot normally. I've been running this VM for some months now. Although both the older and the newer kernel are both installed, neither will boot, except in safe mode. The older kernel had been working just fine. The screenshots below were taken after trying to boot normally. After issuing an acpi shutdown in VBox, I then booted the new kernel in safe mode.



Thinking that the root filesystem may be corrupted, I mounted the installation CD and booted into maintenance mode. fsck is not on the CD. There is about 5GB in use of the 20GB available. Although there is a shared folder between the guest and the host, it is not mounted in failsafe mode. I hope you can see something in the screenshots. I ran journalctl -b, which appears to be showing dmesg. I've skipped most of the screens. I'm showing just the first one and the one where the problems begin, down to the last one.

[claire]

Are you using the packaged video driver or one from upstream?

edit:

That'll teach me to read the whole thing before responding.

[djohnston]

claire,

Thanks for the response. Just in case the question may come up again, there are no outside cources enabled and no outside packages installed. Everything is pure "stock" Mageia3 beta, and always has been. I've been running this long on the beta1 version to see if updates will eventually break something, like happened to me with Mageia2. However, Mageia2 breakage wasn't this bad.

Any logs I should be reading? Are the software update logs kept in /var/log/ somewhere?

[djohnston]

Okay, so I got normal boot working again. Sort of.

1. Jumped into edit mode in the GRUB boot menu by pressing the escape key. Selected the 3.8.5-desktop-1.mga3 entry and edited it. Deleted the splash and quiet entries as well as the resume=UUID entry.

2. Boot got as far as VirtualBox guest additions. Did an ACPI shutdown from the VirtualBox window's menu.

3. Edited the 3.8.5-desktop-1.mga3 entry to start in safe mode. Once there, I did
(a) urpme -a virtualbox
(b) urpme --auto-orphans

4. Selected normal boot for the 3.8.5-desktop-1.mga3 kernel and normal boot was successful.

5. Rebooted and selected 3.8.6-desktop-2.mga3 kernel. Again, normal boot was successful.

6. Because I had no network connection in safe mode, I immediately started the Mageia Control Center and selected Software Management > Update your system. There, I found another kernel and vboxadditions update waiting, among others, because I had the kernel-desktop-latest and vboxadditions-kernel-desktop-latest packages installed. Took a chance and installed all updates.

7. Next reboot was from kernel 3.8.7-desktop-1.mga3. I booted normally, and the boot was successful. I now had three different kernels installed and all were booting normally again. However, I couldn't figure out why my mouse pointer was integrated with both the host and the guest.

8. Started looking in the package list and discovered there are two sets of VirtualBox packages, both metapackages and actual packages. One set is named vbox and the other is named virtualbox. I wasn't sure (I'm still not sure) which set I should have installed.

9. In order to try and sort things out, I uninstalled all vbox and virtualbox packages. Then I uninstalled the two older kernels.

10. Rebooted normally to the newest (3.8.7-desktop-1.mga3) kernel, now the only one installed. So far, so good. Started the package manager and selected the virtualbox-guest-additions package for installation. It required some dependencies which I marked for installation, and installed all marked packages.

11. Next normal reboot failed. It was hanging at the VirtualBox guest additions message again.

12. Rebooted to safe mode. Did:
(a) urpme -a virtualbox
(b) urpme --auto-orphans

13. Normal reboot worked again. Still had integrated mouse pointer with both the host and the guest.

14. Started the package manager. The dependency packages from installing the virtualbox-guest-additions package in step #10 were still installed. The dependency list is:

vboxadditions-kernel-3.8.7-desktop-1.mga3-4.2.12.mga3.i586
vboxadditions-kernel-desktop-latest-4.2.12.mga3.i586
X11-driver-video-vboxvideo-4.2.12.mga3.i586
xrandr-1.4.0-1.mga.i586

(see screenshot below)



Note that all dependency packages are named vbox, not virtualbox, and that I do not have the i586 kernel installed. The installed kernel is 3.8.7-desktop-1.mga3, which is a 686 kernel.

To summarize:

(a) Installing the virtualbox-guest-additions package marked several vbox packages for installation.

(b) Those vbox packages were evidently intended for use with a 586 kernel, which I don't have installed.

(c) Installing the virtualbox-guest-additions package renders the guest VM unbootable, except in safe mode.

(d) Uninstalling the virtualbox-guest-additions package still leaves the vbox packages marked as dependencies installed, even after running urpme --auto-orphans.

(e) Although the guest additions packages intended for a 586 kernel still works with a 686 kernel, there is some jerkiness with the mouse while in the guest VM.

I believe this is a dependency problem, as this situation never arose before now. I'm also still unsure of which packages, exactly, should now be installed for the kernel and VM I am running.

HELP
====
Can/will someone who knows more than me about Mageia's workings tell me:

(a) which packages I should have installed, and
(b) where I can find the package installation logs?

I am more than happy to do any further troubleshooting, posting of logs or anything else someone may want done in relation to this problem.

[claire]

This is reported now as bug 9759 djohnston. Thanks for your follow up comments.

It will be fixed but please feel free to comment on that bug report.

[djohnston]

Thank you, claire. I feel I've been remiss in not signing up for the tester's mailing list, but I'm not sure how. When I click on the Subscribe link here, I am brought to this page which says "In order to subscribe to the list, you should open a account on our account system." How do I open an account? Do I send an email to the address listed for "Contact owners"?

[luigiwalser]

The command "rpm -qa --last" will list all of the packages you have installed, ordered by when they were installed (most recent at the top) and tell you when each was installed. Perhaps this output could help you narrow down which package(s) were upgraded that caused the issue.

[djohnston]

This is reported now as bug 9759 djohnston. Thanks for your follow up comments.

It will be fixed but please feel free to comment on that bug report.

Thanks. I have added to the comments there.

[djohnston]

The command "rpm -qa --last" will list all of the packages you have installed, ordered by when they were installed (most recent at the top) and tell you when each was installed. Perhaps this output could help you narrow down which package(s) were upgraded that caused the issue.

Thanks, Luigi. I have been spoiled by Synaptic's package history GUI.

[claire]

Thank you, claire. I feel I've been remiss in not signing up for the tester's mailing list, but I'm not sure how. When I click on the Subscribe link here, I am brought to this page which says "In order to subscribe to the list, you should open a account on our account system." How do I open an account? Do I send an email to the address listed for "Contact owners"?

You just need to log in at the top of the page. Mageia websites ( forum, bugzilla, wiki, ml. ) all share login details so login with your email address and password where it asks and you should be able to subscribe.

Thanks for getting involved!

[djohnston]

https://bugs.mageia.org/show_bug.cgi?id=9759

After today's update of x11-server and virtualbox packages, all issues have been resolved. It wasn't as easy as just updating and rebooting. I had to uninstall all dkms and virtualbox packages and remove xorg.conf. After doing so and rebooting, I reinstalled virtualbox guest additions, then rebooted. Everything is now working normally again.

Thanks, gentlemen.

[doktor5000]

FWIW, there's another way to query the installed/removed packages history, also listing the program which was used to install them:

Code: Select all

grep -F "[RPM]" /var/log/syslog

or, for packages installed more than 7 days ago, use zgrep to search in the logs which were already rotated:

Code: Select all

zgrep -F "[RPM]" /var/log/syslog*gz

[djohnston]

FWIW, there's another way to query the installed/removed packages history, also listing the program which was used to install them:

Code: Select all

grep -F "[RPM]" /var/log/syslog

or, for packages installed more than 7 days ago, use zgrep to search in the logs which were already rotated:

Code: Select all

zgrep -F "[RPM]" /var/log/syslog*gz

I would never have thought to look in syslog. However, for some odd reason, I don't have one.

Code: Select all

[root@localhost ~]# ls -l /var/log | grep sys
[root@localhost ~]# ls -l /var/log | grep syslog
[root@localhost ~]#

[Ken-Bergen]

I would never have thought to look in syslog. However, for some odd reason, I don't have one.

Code: Select all

[root@localhost ~]# ls -l /var/log | grep sys
[root@localhost ~]# ls -l /var/log | grep syslog
[root@localhost ~]#

Yes in cauldron /var/log/syslog has been replaced by /var/log/journal which is a folder containing the logs which are no longer plain text.
If I remember correctly there are a few posts here on how to read the journal.

[djohnston]

Yes in cauldron /var/log/syslog has been replaced by /var/log/journal which is a folder containing the logs which are no longer plain text.
If I remember correctly there are a few posts here on how to read the journal.

The answer was given by doktor5000. The link he gave is http://0pointer.de/blog/projects/journalctl.html.

Basically, you use the journalctl tool, and it pageinates using less as default.

Holy moly! I see a worrisome entry early on:

Code: Select all

Dec 23 23:51:03 localhost systemd-journal[307]: Allowing system journal files to grow to 1.8G.

1.8GB is pretty big! I'll hafta look into that. Thanks for the response!

[doktor5000]

Yes in cauldron /var/log/syslog has been replaced by /var/log/journal which is a folder containing the logs which are no longer plain text.

Ahh crap, yes, you're right. But seems you can use that interchangeably:

Code: Select all

journalctl -a | grep -F "[RPM]"

[djohnston]

Holy moly! I see a worrisome entry early on:

Code: Select all

Dec 23 23:51:03 localhost systemd-journal[307]: Allowing system journal files to grow to 1.8G.

1.8GB is pretty big! I'll hafta look into that. Thanks for the response!

Easy enough to tweak. Edited /etc/systemd/journald.conf as root. Changed line #SystemMaxUse= to SystemMaxUse=250M. Result:

Code: Select all

journalctl -b | grep systemd-journal
Apr 24 02:55:50 localhost systemd-journal[355]: Allowing runtime journal files to grow to 24.9M.
Apr 24 02:55:50 localhost systemd-journal[355]: Journal started
Apr 24 02:55:53 localhost systemd-journal[355]: Allowing system journal files to grow to 250.0M.

[morgano]

Thank you for the pointer on limiting the size. Historically i have had two issues with filling / by excessive logging.
I wonder if it may be a good idea to have it default set to a sensible value like this?

[luigiwalser]

In Cauldron (Mageia 3) this has been fixed:
- Make default journal default to leaving 15% free space (mga#9921)

[djohnston]

In Cauldron (Mageia 3) this has been fixed:
- Make default journal default to leaving 15% free space (mga#9921)

Hmmm. Well, the previous default was to use 10% of disk size. I agree with morgano. Do we really need months' worth of log files?