[SOLVED] Full disk with df, 72% empty with btrfs fi usage

[john_gibbe]

Hello,
I'm in front of a strange problem on my Mageia5. I have a SSD btrfs partition mount as / which is 28% occupied. After a while system complain the disk is full. I can not figure out what's happening.

# df /
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/sdd2 49G 14G 0 100% /

As you can see df output seems not coherent, with a partition size of 49G, only 14G occupied and a result of 0 available.

# btrfs fi usage /
Overall:
Device size: 48.83GiB
Device allocated: 48.83GiB
Device unallocated: 0.00B
Device missing: 0.00B
Used: 13.71GiB
Free (estimated): 34.90GiB (min: 34.90GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 208.00MiB (used: 0.00B)

Data,single: Size:48.01GiB, Used:13.10GiB
/dev/sdd2 48.01GiB

Metadata,single: Size:835.00MiB, Used:623.83MiB
/dev/sdd2 835.00MiB

System,single: Size:4.00MiB, Used:16.00KiB
/dev/sdd2 4.00MiB

Unallocated:
/dev/sdd2 0.00B

It looks it's not the classical trouble of reserved space as only 208MiB are blocked.

# btrfs filesystem show /
Label: none uuid: a0502927-c276-455e-934a-7688108e9093
Total devices 1 FS bytes used 13.71GiB
devid 1 size 48.83GiB used 48.83GiB path /dev/sdd2

# du -sx /
14G /

Any ideas?

[doktor5000]

Hi there, please post the output as root of

Code: Select all

/bin/mount
lsblk -o NAME,FSTYPE,SIZE,LABEL,MOUNTPOINT
df -Pm
du -mx /|sort -rn|head -5

I'd also suggest checking for files that have been deleted but where some process is still keeping a filehandle open so the space is not freed up. See e.g.

Code: Select all

lsof /| grep '(deleted)'

or

Code: Select all

lsof -a +L1 /

[john_gibbe]

Hello Doktor5000, here are the result of the proposed commands

Code: Select all

# mount | grep sdd
/dev/sdd2 on / type btrfs (rw,noatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/sdd1 on /boot/EFI type vfat (rw,relatime,fmask=0000,dmask=0000,allow_utime=0022,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro)
/dev/sdd4 on /home type btrfs (rw,noatime,ssd,space_cache,subvolid=5,subvol=/)


lsblk for sdd:

Code: Select all

# lsblk -o NAME,FSTYPE,SIZE,LABEL,MOUNTPOINT
sdd           931,5G       
├─sdd1 vfat     299M       /boot/EFI
├─sdd2 btrfs   48,8G       /
├─sdd3 swap     3,9G       [SWAP]
└─sdd4 btrfs  878,5G       /home


Code: Select all

# df -Pm | grep sdd
/dev/sdd2                   49999   14252          0     100% /
/dev/sdd1                     299       1        299       1% /boot/EFI
/dev/sdd4                  899567  756107     142278      85% /home


Code: Select all

# du -mx /|sort -rn|head -5
13856   /
10844   /usr
4585    /usr/lib64
2898    /usr/share
2267    /var


Code: Select all

# lsof /| grep '(deleted)'
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
kded4     31066             john   10r   REG   0,23  2207947 1079299 /var/tmp/kdecache-john/ksycoca4 (deleted)


Code: Select all

# lsof -a +L1 /
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
kded4   31066 john   10r   REG   0,23  2207947     0 1079299 /var/tmp/kdecache-john/ksycoca4 (deleted)


[doktor5000]

Could you please edit the post and add the complete output of mount again? I deleted the second post as it was the same as this one, but oversaw that it misses some of the mount output ...

[john_gibbe]

Hello again.
The important thing was

Code: Select all

# btrfs filesystem show /
Label: none uuid: a0502927-c276-455e-934a-7688108e9093
Total devices 1 FS bytes used 13.71GiB
devid 1 size 48.83GiB used 48.83GiB path /dev/sdd2


it meant that for the partition sdd2 the file system used only 13.71 GiB but btrfs used 48.83GiB of 48.83GiB on that partition.
As my system was not anymore bootable, I booted on a Magiea5-Live USB key and mounted ssd2 on /mnt. Then I managed to free a little space by deleting some /var/log/ files. This allowed me to apply with success the following command:

Code: Select all

# btrfs balance start -dusage=2 /mnt
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=2
Done, had to relocate 1 out of 55 chunks


I repeated this command several times while incrementing slowly -dusage= at the begining (started with step of 1) and more quickly after several chunks have been relocated (step of ten) until I reached 100. I have now:

Code: Select all

# btrfs filesystem show /mnt
Label: none  uuid: a0502927-c276-455e-934a-7688108e9093
        Total devices 1 FS bytes used 13.41GiB
        devid    1 size 48.83GiB used 15.07GiB path /dev/sdd2


Wonderfull, my system is now able to boot again!

I read that with kernel > 3.19 there is no need to do that manually. I run now this kernel:

Code: Select all

# uname -a
Linux PC-JOHN 4.4.30-desktop-2.mga5 #1 SMP Fri Nov 4 19:17:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


so as I understand it should not be necessary to do that and it should be automatic.
Is something wrong the way sdd2 is mounted?

[doktor5000]

Would be helpful if you would add how long you have been using that installation, and how you used the filesystem before this occured.

From a quick comparison, can't see much differences to what I have, although you seem to be using subvolumes ... did you create those partitions and filesystems yourself, or how did you do it?

I believe (although not 100% sure, installed quite some time ago) those are pretty much the default options, used by the Mageia 5 installer for btrfs on an SSD:

Code: Select all

[doktor5000@Mageia5]─[17:54:04]─[~] grep btrfs /etc/fstab
UUID=799cdaf8-b6af-4299-a6c9-c934027e3bdc / btrfs noatime,discard,ssd,autodefrag,compress=lzo,space_cache,ssd_spread 1 1
UUID=a6d498ba-009f-4ee0-8fee-1460f2622dba /home btrfs noatime,discard,ssd,autodefrag,compress=lzo,space_cache,ssd_spread 1 2

and so far I've not had any issues with it, since the installation, even when heavily using it (package building/compiling ...)

[john_gibbe]

Hello,
As far as I remember I set this SSD when Mageia 5 was issued and I chose to install it instead of upgrading Mageia 4. As I wanted more space for / than usually given by default, I chose the expert mode to partition. Now my fstab is as follow:

Code: Select all

# Entry for /dev/sdd2 :
UUID=a0502927-c276-455e-934a-7688108e9093 / btrfs ssd,noatime 0 0
# Entry for /dev/sdd1 :
UUID=945D-4047 /boot/EFI vfat umask=000,iocharset=utf8 0 0
# Entry for /dev/sdd4 :
UUID=a9169ca2-ca90-4afb-9e7e-8a36e6e23286 /home btrfs ssd,noatime 0 0
none /proc proc defaults 0 0
# Entry for /dev/sdd3 :
UUID=296a8880-8873-4e36-acbc-4ce96bd9f932 swap swap defaults 0 0
# Entry for /dev/sda6
UUID=e367901c-0494-4d2d-a691-e6aa0225ce09 /home/usr_fff/svgd btrfs noatime 0 0
none /tmp tmpfs size=8G 0 0



Apparently I missed some options such as autodefrag or discard. In your fstab fifth field (fs_freq) and sixth field (fs_passno) are not null. I believe to remember to have read that btrfs doesn't need to be checked at boot time and these should be 0 0.
What do you think?

[doktor5000]

Apparently I missed some options such as autodefrag or discard. In your fstab fifth field (fs_freq) and sixth field (fs_passno) are not null. I believe to remember to have read that btrfs doesn't need to be checked at boot time and these should be 0 0.
What do you think?

To be honest I'd need to recheck for autodefrag and discard.
I don't remember explicitly putting discard there, as usually I use fstrim once every few months but that's only a performance thing for the underlying physical disk and should not affect the filesystem fill state in any way.
For autodefrag I'm pretty sure that's there by default, but I'll check my notes on customization after installation.

For the fsck options, those are the defaults and I don't think it hurts to have those. Although I don't remember any instance where fsck ran during bootup and found some issues, so you may be correct that it's not required.
Only actual definitive information I found on this topic is https://btrfs.wiki.kernel.org/index.php ... fsck.btrfs which seems to match with what you found, explicit fsck is not necessary during boot.

[john_gibbe]

Ok, I will let fstab as it is and will check more often how this partition will evoluate.
If in the future used space diverge too much from fs space I will try the discard option in fstab or the fstrim in a cron file.
Anyway at this time my problem is solved.

Thanks a lot for your help doktor5000.

[doktor5000]

If in the future used space diverge too much from fs space I will try the discard option in fstab or the fstrim in a cron file.

Please read again. There's no relation between discard/trim and used disk space in a filesystem, discard/trim relates to the internal mechanisms in the block device underneath the filesystem (flash-based storage needs to be deleted before it can be written to again, which is what trim/discard does as most controllers default lazy deletion). If there was a relation, I'd consider that a bug.
In the worst case, if you don't use trim/discard, then the write performance of that SSD would be slightly below the optimal values, and that's about it.

[ozky]

There is also upstream patch to fix this df problem take a look this Mageia bug and this patch.
My own laptop suffer this problem that some rpm failed to install and df show disk is full.

https://bugs.mageia.org/show_bug.cgi?id=19728#c17

http://www.phoronix.com/scan.php?page=n ... 4.8-ENOSPC

https://patchwork.kernel.org/patch/9411743/