Massive excessive mystery disk usage

[Xhermin]

Lately, my Mageia 3 system (2xP4@3GHz, 1GB RAM, 2 GB swap) is becoming nigh-unusable. After behaving for several hours of usage (websurfing, facebook, youtube, gmail, yahoo, all common sites) It sits for tens of minutes, headslamming with massive disk activity, gui effectively halted, can't even ctrl-alt-backspace or ctrl-alt-delete, even switching to a text-only screen (tty5) is visibly slowed. Half an hour now, I'm typing this on my laptop while the desktop just grinds. I don't even know how to diagnose the problem because I'm not sure I can trust what Ksysguard (or kde4's successor of it) or "top" command is telling me. My desktop panel widget tells me (while it still works, it froze a while ago) that I am at 85-100% CPU utilization, yet the Process Viewer (and top command in tty5) insists that maybe about 10% of cpu is being used. I've seen this sort of discrepancy across several kde4 distros, and really makes me long for kde3. Anyway, back to current, watching top for a few minutes gets occasional spikes of "kswapd0" but I don't know if that's cause or effect. Can't capture the screen with the machine in its current state, so I'll transcribe the current status of the gui System Activity screen (which just appeared, I clicked on it minutes ago). I'm rounding memory use figures.

kswapd0 (root) (remainder of line blank)
chrome (myself) cpu%=disk sleep, mem=40M, shared = 4M ; Page(s) unresponsive
udisksd (root), cpu%=(blank), mem=1M, shared=152K
mplayer (myself) cpu%=(blank), mem=10M, shared=296K
plasma-d... (myself) cpu%=disk sleep, mem=69M, shared=6M
dolphin (myself) cpu%=disk sleep, mem=12M, shared 3M ; %podcast_directory%
smplayer (myself) cpu%=(blank), mem=8M, shared 2 ; %podcast_mp3_being_listened_to_when_system_froze%
(...)

everything else visible on the System Activity list are root-owned system files, no cpu or memory usage, save for udisks-d... (mem=664K, shared=112K), acpid (mem=100K), upowerd (mem=804K), and flush-8:0 (cpu%=disk sleep)

Scrolling down the list to see the bottom is impossible. This typed, I am now going to halt the offending machine the only way I can -- power switch hard shutdown. It'll come back up ok (it has before)....but for how long?

I am at wit's end here, be happy to provide additional info, but if I have to really fight with it to fix it, I'll reinstall. Probably to a different distro.

[Xhermin]

Like another poster, I do like Mageia, I really do. But I like machines that work properly, too.

Hardware should largely be irrelevant. I've run the affected machine under Mageia 2 and upgraded to Mageia 3 when it came out with no such difficulties until now. It's 1am my time, need to sleep, Wll provide requested info later.

[doktor5000]

Your issue was mostly about disk usage, did you take a look what made up for the large amount of harddisk activity?
Please install iotop and when it happens next time, please take a look at what the culprit is.

When the disk activity started, where you using the system actively or was it idle before the disk activity started?

[jiml8]

Take a look in /var/log/journal.

As configured, the journal daemon is writing to your hard drive without limit, and WILL fill your drive. There are some config settings that are supposed to manage that, but they don't seem to be working correctly (at least, not for me). I've been visiting that directory every couple of days and deleting everything, then restarting journald. I have been thinking about just disabling it; I use syslog anyway.

[doktor5000]

One could use limits for that, those should be effective, otherwise that's a bug:

Code: Select all

[doktor5000@localhost SPECS]$ grep -i size /etc/systemd/journald.conf
#SystemMaxFileSize=
#RuntimeMaxFileSize=

Check

Code: Select all

man journald.conf

for that.

[jiml8]

I have set the limits and they don't seem to be working. I have not reported it as a bug because I am not yet convinced I did it right - though it seems easy enough - and because my shorewall installation has been reporting huge quantities of packet information into the log files. I run a TOR relay, and I host some freely distributable bittorrent files, and the consequence is that I have lots of traffic here - and shorewall wants to report about all of it.

Just today, I have been studying the shorewall docs in order to tame it and stop all that logging, and I think I have it sorted out. So now I will watch system-journald for awhile to see if it is now behaving itself.

In any case, this is not to hijack this thread, but to warn OP that the default install of Mageia incorrectly configures system-journald and allows logs to grow without limit. This has been reported as a bug, I believe, but it is certainly a likely place for OP to start looking for his problem.

[doktor5000]

That's why I've proposed to take a look with iotop what produces all the disk accesses.

[Xhermin]

As I recall, I was just listening to podcasts saved as mp3 to my harddrive, catching up on backlog. Checking a couple emails via chrome, not doing a lot active/interactive with the machine, mostly just sitting and listening.

I will install iotop when I get a chance, I'm going to be very busy for the next few days, long work hours, may not get to this right away. Keep thread open please.

[ITA84]

I have set the limits and they don't seem to be working.

I recall that being a problem on some journald versions: I had set a size limit in journald.conf, and after passing the limit it'd go into an infinite loop while repeatedly logging some error message (after a reboot it'd stop, but the size limit wasn't enforced anymore). Can't remember which version it was (it's gone now on Cauldron), nor if it's been patched in Mageia 3.

[jiml8]

I recall that being a problem on some journald versions: I had set a size limit in journald.conf, and after passing the limit it'd go into an infinite loop while repeatedly logging some error message (after a reboot it'd stop, but the size limit wasn't enforced anymore). Can't remember which version it was (it's gone now on Cauldron), nor if it's been patched in Mageia 3.

I did see this exact behavior just a few days ago. Error messages appeared on every console and the KDE UI no longer would respond. I could not make systemd-journald stop...I was typing blind since the error messages swept away whatever I typed, so I could not check for typos or syntax errors and could not refer to the manpage. I finally issued a telinit 1 and journald was shut down. I then deleted everything in /var/log/journal (which was way larger than it should have been) and issued a telinit 5. Things are working again.

[ITA84]

If the error message was something like "Sleeping for xxxxxxxxus", I think it was fixed in systemd v199; they fixed the limits as well, but I don't know in which version.

[Xhermin]

Okay, it's clearly linked with web browsing, an starts happening now almost as soon as I got to a page on the www.

With iotop, I'm seeing firefox occasionally itself popping up to the head of the class with a commandline of:

Code: Select all

firefox --sm-config-prefix /firefox-xekqNc --sm-~d  %long string of numbers%


and the firefox plugin-container, but too quickly to really catch the text. flash plugin maybe?

But what I'm seeing much more of is mysqld, referencing:

Code: Select all

 ~/.local/share/akonadi...

roughly a dozen instances of this program. Akonadi itself is towards the bottom of the page.

All this ONLY when I'm actively online, surfing.

There are a few others I don't immediately recognize, like "aria2c" and "utpd"

[ITA84]

Maybe you could try and reproduce the problem with a different user: if you don't notice the same issue, then you could just clear user settings for Firefox and/or Akonadi (after backing them up) and see if the problem goes away.

[jiml8]

I have not noticed disk usage getting large due to either flash or firefox, though both of those are behind memory leaks I have observed. The latest version of flash seems to have greatly reduced this problem.

Your akonadi might be indexing the cached stuff that firefox downloads. This would have an impact on your disk usage but I'm not sure how much. I do know that the database associated with akonadi gets rather large.

Try this. From your home directory, enter the command "find -size 10M" (without the quotes). This will cause a listing of all files in your home directory that are greater than 10 Megs in size. You can adjust this command for size by adjusting the size paramter. This should let you zoom in fairly quickly on the largest files in your home directory.

To find directories that are getting large (as in: many many files are being written there...) use "find -type d -size +100k" where again you can adjust size as appropriate.

[filip]

I do know that the database associated with akonadi gets rather large.

I turn off indexing as database grew over 10GB. I found out that after "Massive excessive mystery disk usage" of hourly rsnapshot backup script.

From your home directory, enter the command "find -size 10M" (without the quotes).

I would use "du -x -m --max-depth=1 /home | sort -g" for that. One of Doktor5000's excellent tips.

[jiml8]

I would use "du -x -m --max-depth=1 /home | sort -g" for that. One of Doktor5000's excellent tips.

This will fail to find it if the large directory is a subdirectory more than one level under /home.

[doktor5000]

Nope, it will show it, but actually the command you posted and the one filip mentioned have a different scope.
And the one filip mentioned can also be shortened to

Code: Select all

du -smx ~/* | sort -rn

@filip: If you use that, you should be aware about e.g. the differences of du and df.
jiml8 is right as find will also process hidden directories, and can easily differentiate between files and directories.

[Xhermin]

Just so you folks understand, it's not file SIZE causing the problem, so much as I know. Free space on my hd's are about where they should be (20GB boot/swap drive, 250GB mounted as /home). It's CPU utililization, where ~10% is for program use (shown in System Activity, ksysguard, or top) and the remainder is disk I/O, visible only using iotop, which shows multiple threads attempting very high (99%) cpu utilization, and the (dual-core) processor (ancient, I know...) can no longer keep up with the desktop, mouse clicks take whole minutes to register, the whole system just bogs down while hard drive accesses are grinding away like mad.

I looked at the akonadi FAQ, per their answer for high cpu usage, disabled indexing in krunner; no change in my system behavior.

Haven't had time to go prying thru the /var/log/* files, need to fall over and sleep now. Stay tuned, and thank you for your patience.

[jiml8]

Well, that is a horse of a different color.

When I review your first post, I see that I did in fact misinterpret what you were asking.

Generally, the problem you describe, where your processor is waiting for long periods of time to read or write, heralds a hard drive problem. You need to check in /var/log/messages (though I suppose this information will also be found in /var/log/journal/blah blah, but I haven't used journald long enough to encounter those things) looking for evidence of I/O errors with your hard drive.

As I think I mentioned earlier in this thread, I myself have on one occasion had a problem where KDE became nonresponsive apparently because systemd-journald had gone nuts and was in an infinite loop reporting an error and failing to write. The error (IIRC) was something about the string to write being too long (which seems nonsensical to me) along with a failure to open the log file. I forced the system into init level 1 which stopped a lot of services. I then deleted all the contents of /var/log/journal and went back to init 5. The problem has not reoccurred since, and I have no idea what caused it. I have, however, subsequently blasted my system drive with compressed air.

Now, about 6 weeks ago I did have an exercise where one of my hard drives was throwing errors. This was the drive that has /home on it, so it was a significant problem. I wound up shutting down the machine, removing that drive, and cleaning all the dust off it. There was a little blob of dust on the controller card such that several interface pins could have been affected. After putting the drive back in place and reconnecting it, I have had no further trouble.

[andrew-lohmann]

If you have deleted and created new partitions such as reusing an old hardisk used previously for Windows the operating system will have to identif bad sectors and mark them bad. I have a hardisk which is fast and reliable but it has an unsually high number of surface errors which probably would take a few days to identify and mark if I did a full format to mark bad sectors.

I have been using this hard disk to see how Linux handles this issue. It handles it well. The thing is if you change to another linux use the existing partitions.

If bad sectors turn up randomly then that is time to discard the harddisk, cable or something else. I had a bad mainboard this left the hard disk with some permant surface errors and a warning flag set (which has subsequently cleared as the errors stopped arrising)