[SOLVED] My expanded /var - full again!

[Myles]

Hi all.
I had this exact same problem about a year ago. In that problem my /var was only 6.8Gb in size. After I re-sized partitions (NTFS and /var) my /var was (and is) 14Gb in size. It too has completely filled up.
I discovered this problem this morning when I tried to install an update, it failed as /var was full.
So I'm at a standstill as I can't install anything, as any update gets temporarily stored in /var. also I can't print anything for the same reason.

Using Dolphin I found this.
The main culprits are 3 folders I have found so far:
/var/log - 3.9Gb
I can't see anything specific in /var/log taking up 3.9Gb, but there are some folders I can't access via Dolphin.
/var/lib/plexmediaserver - 3Gb (I expected this but I need to keep that)
/var/spool/mail/ there is something called "root" that is 3.7Gb big.

I'm not sure what other type of info you might need to help me out with this.

So what is going on, /var is twice the size now. Why are /log and /spool so huge?????
I don't want to have to go through the rigmarole of re-sizing all the time, that seems ridiculous. I obviously don't have something set up properly to manage data going into /var, but don't know what that is.

I should say I am using Dolphin to do this, I'm not a CLI-type of guy.

As always, any help greatly appreciated.

[magic]

Well, /var/log contains the log files so you can look into deleting the older saves and maybe setting some limits on sizes for the future. (I've no experience doing this)

/var/spool/mail holds mail messages - the root file holds the root mail messages (probably mostly security audits) - you can read them in a terminal with the mail command.
If you are sure that there are no messages that you are interested in you can just empty the file. I don't know if there is an easy way to do this graphically but, from the terminal as root:

Code: Select all

> /var/spool/mail/root

(yes, it has a '>' at the start) should do the job.

---
Out of interest - is there a particular reason you have a separate /var partition?

[doktor5000]

Well, /var/log contains the log files so you can look into deleting the older saves and maybe setting some limits on sizes for the future. (I've no experience doing this)

journal files should not be manually deleted, at least if they're not orphaned/corrupted (IIRC those end with a ~).

@Myles:
You can easily shrink down journal logs to a certain amount of MB or to a certain amount of days. Have a look at the journalctl manpage
for --vacuum-size and --vacuum-time.

If you only have limited space, you might want to consider setting sensible limits regarding max size in /etc/systemd/journald.conf.

[Myles]

I thought I had posted this late last night, but obviously not as I can't see it anywhere.

I have a separate /var because that was one of the recommendations that I found when I first set up my linux system.

doktor,
All fot rhe normal system logs contained in root of /var/log are all okay. They are rotating - there is a current log plus 4 previous logs. The total space used by these logs is about 800Mb.
The problem in /var/log lis within one or more of these other folders I can't access, as under Dolphin most (but not all) they show a lock on the folder icon and won't allow me access - I get a msg "Could not enter folder /var/log/hp" or whatever the folder is, even if I just try to get the properties of the folder.

Code: Select all

These folders are:
/var/log/cron
/var/log/daemons
/var/log/hp
/var/log/kernel
/var/log/mail
/var/log/ppp
/var/log/samba
/var/log/security
and
/var/log/sudo-io

Is there some sort of CLI command that I can execute under sudo or root that lists the folder size and the size of all files within that folder?

Also, is it safe to just clear the /var/spool/mail/root file to give me some breathing space as was suggested?

Anyway at the end of all of this I need a way of going forward and managing the space so that /var doesn't fill u again.

[doktor5000]

Is there some sort of CLI command that I can execute under sudo or root that lists the folder size and the size of all files within that folder?

Yes, and it's also pretty easy to remember, although at first the output might be slightly confusing to read

Code: Select all

du -mx /var | sort -rn | head -50

will show the 50 biggest folders under /var filesystem, sorted descendingly by size in MB.

And yes, you can just null /var/spool/mail/root as mentioned above. Although on a default installation mails aren't forwarded there, so you probably have installed any MTA (mail transport agent) at some point in time.
Otherwise those would have ended up in the file /dead.letter

[Myles]

doktor,
Thanks for that. I executed the command under sudo (and root) but it still doesn't show which folders - contained in /var/log are taking up the 3.9Gb of space. I'm presuming it's one or more of these "protected/locked" folders that I can't access. So why doesn't sudo or even root get access to list these?

Regarding the /var/spool/mail folder - yes I installed at least one mail program (sendemail I think) a while back but I haven't been using it. I was going to set it up to send me emails if a LuuckyBackup job had errors. I'm still hoping to set it up. So this my be the cause of that. Odd though it goes to a file named "root".
Anyway I just cleared that file out and now have some breathing room. I'll have to keep a closer eye on what's happening in /var in regard to space.

As usual, thanks all for the help, especially the ever reliable doktor.

[doktor5000]

I executed the command under sudo (and root) but it still doesn't show which folders - contained in /var/log are taking up the 3.9Gb of space. I'm presuming it's one or more of these "protected/locked" folders that I can't access. So why doesn't sudo or even root get access to list these?

Could you post the output please ?

Regarding the /var/spool/mail folder - yes I installed at least one mail program (sendemail I think) a while back but I haven't been using it. I was going to set it up to send me emails if a LuuckyBackup job had errors. I'm still hoping to set it up.

Delivering mails to local user accounts doesn't require any setup, that is standard functionality once you installed an MTA.

[Myles]

doktor,
Here is the output from the command under root:

Code: Select all

[myles@Bilbo ~]$ su root
Password:
[root@Bilbo myles]# du -mx /var | sort -rn | head -50
9955    /var
5648    /var/log
3873    /var/lib
3391    /var/lib/plexmediaserver
3275    /var/lib/plexmediaserver/Library/Application Support/Plex Media Server
3275    /var/lib/plexmediaserver/Library/Application Support
3275    /var/lib/plexmediaserver/Library
1710    /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata
1441    /var/log/journal/badb2719d00f4b4697ade2f7fd49c6ba
1441    /var/log/journal
934     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows
803     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media/localhost
803     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media
777     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies
620     /var/log/daemons
568     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache
514     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder
430     /var/cache
393     /var/log/security
217     /var/cache/urpmi/rpms
217     /var/cache/urpmi
214     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/b
175     /var/cache/dnf
172     /var/lib/mysql
162     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support
150     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases
147     /var/lib/rpm
116     /var/lib/plexmediaserver/.luckybackup-snaphots
115     /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/f
86      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/c
82      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/3
78      /var/log/kernel
77      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/8
75      /var/lib/plexmediaserver/.luckybackup-snaphots/20180114100002/plexmediaserver/Library/Application Support/Plex Media Server
75      /var/lib/plexmediaserver/.luckybackup-snaphots/20180114100002/plexmediaserver/Library/Application Support
75      /var/lib/plexmediaserver/.luckybackup-snaphots/20180114100002/plexmediaserver/Library
75      /var/lib/plexmediaserver/.luckybackup-snaphots/20180114100002/plexmediaserver
75      /var/lib/plexmediaserver/.luckybackup-snaphots/20180114100002
73      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/6
73      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/1
72      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/b/aee43710172df388319661fe9feea9aa76fe33a.bundle
70      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/9
70      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/7
66      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/a
66      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/2
64      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/f
63      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/3
62      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media/localhost/e
60      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies/4
59      /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media/localhost/d
[root@Bilbo myles]#

[doktor5000]

Well you got 5.6GB below /var/log so that's the first I'd take a look at. You said logrotate is working fine, I'd say no as otherwise logs except /var/log/journal would be compressed and only take a few MBs ...
I've also provided how to shrink the size of journal log, have a look at that again.


Another option, which might be more easily readable, is to get the summary sizes

Code: Select all

du -smx /var/* | sort -rn

[Myles]

doktor,
I can see that /var/log is 5.6Gb in size as a result of the command you posted. And I have no idea what the heck is going on, I obviously made some major mistake. I was sure that the system "log-rotated" logs only took up about 800Mb. I have just done a properties on the total of ALL the log files (excluding folders) at the root of /var/log and the total is 3.1Gb! The size of the /var/log/journal folder is 1.5Gb! So there is 4.6Gb of that 5.6Gb space it seems!

Also on a side note, there are 12 logs named "file:///var/log/Xorg.pid-xxxxx.log", where "xxxxx" is a string of digits - a process Id I presume. Most of these date back to last year (8), and the rest from this year, all with a size of 0B. What would these be, and is it okay to delete them?

As for the Journal logs and the system "log-rotated" logs I'll have to implement your suggestions to manage the size of the them.

[doktor5000]

Also on a side note, there are 12 logs named "file:///var/log/Xorg.pid-xxxxx.log", where "xxxxx" is a string of digits - a process Id I presume. Most of these date back to last year (8), and the rest from this year, all with a size of 0B. What would these be, and is it okay to delete them?

Asking the other way around, if they're 0B in size, what would you need those logs for?
I'd say they might be leftovers from an X server crash or from an unsuccessfuly start.

[omnio]

This happened to me too. I installed Mageia leaving pretty much free space on the partition but after a couple of months it got filled up. You would think that logrotate just takes care of everything by default but things got problematic and rather unpredictable since systemd came around.

[doktor5000]

And what in particular filled up your /var ?

And logrotate only takes care of what is configured under /etc/logrotate.d/ but e.g. journald has it's own configuration.

[omnio]

And what in particular filled up your /var ?

things got problematic and rather unpredictable since systemd came around.

=> journald

[doktor5000]

And logrotate only takes care of what is configured under /etc/logrotate.d/ but e.g. journald has it's own configuration.

[omnio]

things got problematic and rather unpredictable since systemd came around.

[doktor5000]

And how does the help the OP, while you are hijacking this thread? Also, apart from mentioning that it's "problematic" and "rather unpredictable" and repeating yourself, what are your particular points about it?
Did you check your journald configuration and adjust it to your liking?

[omnio]

Did you check your journald configuration and adjust it to your liking?

Exactly my point. Adjusting journald configuration isn't something that users should be forced to do unless they followed some very exotic install scenario. And I put it gently, actually they shouldn't be forced to do it anyway -- there already is a log rotating mechanism out there (called logrotate), systemd shouldn't be an exception and should have its logs handled like the rest of the logs. This is why I stated that "things got problematic and rather unpredictable since systemd came around". I hope I made myself clear now.

[doktor5000]

This is why I stated that "things got problematic and rather unpredictable since systemd came around". I hope I made myself clear now.

No you didn't. Do you expect that systemd/journald will change its default configuration to appeal to your setup - without even known what's wrong in particular about that setup, as you didn't yet mention that.
So this whole part in this thread is nearly pointless. You could also have abbreviated that as "yadda yadda I don't like systemd/journald, and I changed journald configuration to my liking" and it would have contributed exactly the same.
FWIW I've never been "forced" to change default journald configuration, and also my /var or / never ran full because of journald. Yet you do generalize your issue as if everybody would be affected by this. So the question still stands, what's special about your and OP's setup so that /var is getting filled up by journald.

[omnio]

"I don't like systemd/journald"

Yes, you are right, I don't like systemd.

"yadda yadda"

You are wrong, what you considered to be "yadda yadda" were logical and reasonable statements.
The problem is, when you spend too much time buried into technical details and helping others (and you're doing a fine job with that btw) you may lose the big picture.

[doktor5000]

you may lose the big picture.

I don't think so. And if people say "xyz is bad, it needs to change" without mentioning how it should actually change in particular, I guess discussion is kind of futile.