[Solved] system has become VERY swappy

This forum is dedicated to advanced help and support :

Ask here your questions about advanced usage of Mageia. For example you may post here all your questions about network and automated installs, complex server configurations, kernel tuning, creating your own Mageia mirrors, and all tasks likely to be touchy even for skilled users.

[Solved] system has become VERY swappy

Postby jiml8 » Feb 12th, '21, 20:51

My last updates seem to have returned my consoles to me, and seem to have taken care of another problem apparently in dbus. So that's good.

But now my system loves my swap. I have turned vm.swappiness down to 1 in order to minimize swapping, but at this time something in excess of 6 GB of my swap is in use. Specifically,
Code: Select all
root@dadsbox:jiml> sysctl -a | grep swapp
vm.swappiness = 1

root@dadsbox:jiml> free
              total        used        free      shared  buff/cache   available
Mem:       32784036     6392636     1513528    19836636    24877872     6049716
Swap:      36909400     6666660    30242740


Other than the update, nothing at all has changed in the system. My normal usage pattern continues, and there should be enough RAM to keep the swap to a minimum. I normally expect the system to be using less than 1GB of swap.

This usage of swap is causing noticeable slowdowns when I access a VM that I have not accessed for a few hours. I do have the swap available, just in case, but this is ridiculous.

Does anyone have any ideas?
Last edited by jiml8 on Feb 17th, '21, 00:31, edited 1 time in total.
Posts: 1165
Joined: Jul 7th, '13, 18:09

Re: system has become VERY swappy

Postby doktor5000 » Feb 13th, '21, 03:43

Did you check what is actually using swap currently, with something like smem or similar ?
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
Posts: 16463
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: system has become VERY swappy

Postby jiml8 » Feb 13th, '21, 18:42

I was not familiar with smem, but that seems like a very useful tool.

The big culprits are chromium, akonadi, and vmware. One of my VMs is the biggest single user of swap. I don't know how that could be, but I am looking into it.

I upgraded to vmware workstation 16 a couple of months ago, but this assault on my swap did not start until after the latest mageia updates that I installed. But perhaps there is some new vmware setting somewhere that is allowing this.
Posts: 1165
Joined: Jul 7th, '13, 18:09

Re: system has become VERY swappy

Postby jiml8 » Feb 17th, '21, 00:31

There is a global setting in vmware workstation to select how swap is used with VMs. You can not allow any VM memory to be swapped, or allow some to be swapped, or allow swapping to occur aggressively.

For many years, I have had my workstation installation set to allow some memory to be swapped, and this has worked well.

No longer.

Now, I have changed the setting to do not allow any VM memory to be swapped, and my swap usage has dropped, while my VM performance has improved.

Something has changed in the most recent kernel, and likely something is also different in Workstation 16 vs Workstation 14 (which was the upgrade I made...14 to 16), and the combination made performance crappy with workstation allowed to swap memory.

Now, memory is getting swapped anyway when it needs to happen, but the host kernel is doing it, not Workstation.

So, I don't know what the details are but the setting change in Workstation has made some big improvements. Swap usage is still much higher than I am used to seeing (right now, about 4.5 GiB in use) but it is well down from what I was seeing before the setting change.
Posts: 1165
Joined: Jul 7th, '13, 18:09

Re: [Solved] system has become VERY swappy

Postby jiml8 » Feb 21st, '21, 00:27

While my configuration change in vmware has restored acceptable operation to the overall system, I am still seeing swap usage creeping up steadily though not rapidly, even as the system's behavior does not seem swappy.

So, while I am not sure there is a problem here, something has definitely changed and I would very much like to know what it is.

I seem to be zeroing in on kcompactd, which is a somewhat new feature that my monitoring tells me is running from time to time, and sucking up a lot of processor when it does run.

Some digging turns up this article:

which describes what is going on.

Again, I am not sure there is a problem here because I do have plenty of swap. Nevertheless, does anyone else here have any comments about this?
Posts: 1165
Joined: Jul 7th, '13, 18:09

Re: [Solved] system has become VERY swappy

Postby jiml8 » Mar 18th, '21, 23:09

I continued to be unhappy with overall system performance, notably in virtual machines.

And I kept seeing kcompactd0 appearing high in my process list, using a lot of CPU.

So, I looked in sysctl, and found this tunable:
Code: Select all

I changed that value to 10, and the difference in system performance is amazing. Much less CPU usage, and the issues I was having with my VMs (particularly the heavily utilized VMs) seems to have vanished.
Posts: 1165
Joined: Jul 7th, '13, 18:09

Re: [Solved] system has become VERY swappy

Postby morgano » Mar 19th, '21, 00:13

Good find!
Thanks for the update.
Mandriva since 2006, then Mageia since 2011 at home & work. Thinkpad T43 & T400, Dell M4400, Acer Aspire 7. Workstation using LVM, LUKS, VirtualBox, BOINC
Posts: 745
Joined: Jun 15th, '11, 17:51
Location: Kivik, Sweden

Re: [Solved] system has become VERY swappy

Postby jiml8 » Apr 16th, '21, 18:39

This vm.compaction_proactiveness continues to be a problem for me, though it is a greatly reduced problem.

I have adjusted the value down to 1. I just did that; it has been at 5 for the last couple of weeks.

Usually the system is OK, but recently I had an rsync that ran for almost 48 hours, syncing (with hardlinks) between 2 multi-terabyte directories (this was part of a crash recovery where a filesystem was so badly damaged I had to rebuild it).

Given that this rsync had to handle lots of hard links, it wound up occupying a huge amount of RAM, and began impacting my system. Swap usage got as high as 15 GB, and that gawdawful kcompactd0 kept showing up - and every time it showed up, it partly or fully stalled my system until it was either done or gave up. Its attempts to "optimize" my memory usage are making the system hard to use.

I have checked, and neither Mint nor OpenSUSE have this thing compiled in, and I think Mageia would do well to compile it out. It drives swap usage to levels well beyond what is necessary, and in my experience interferes far more than it helps. If I had the option to disable it I would cheerfully do so.

For reference, my system has 32 GB RAM and I for the last few days I have been running 4 VMs along with the host. One is a Mint 19 VM that has 4 GB RAM assigned, one is a Mint 20 VM with 4GB RAM assigned, one is an OpenSUSE Leap 15.2 with 4GB RAM assigned, and one is a FreeBSD 8.4 with 2GB assigned. Thus, I have 14 of my 32GB assigned to VMs. Allow 10% overhead for the VMs, then round up, and half my memory was dedicated to VMs.

The rsync was showing usage of around 3 GB at its peak, and for part of the time I was running bitcoin core, which sucked up another 4 GB or so (I shut it down because I didn't really need it and it was impacting things heavily).

I also was running Firefox with dozens of tabs open and chromium with about 10 tabs open.

Now, chromium is a gross memory hog, but even so there is no way this configuration should have hit 15 GB swap, and that kcompactd0 should not have been crapping all over me.

This configuration is not unlike the configuration I have been running for years. It is even a bit lighter because I did not have any Windows OS's open. I have never had these problems until kcompactd0 appeared. I would be very happy if it disappeared.
Posts: 1165
Joined: Jul 7th, '13, 18:09

Re: [Solved] system has become VERY swappy

Postby magic » Apr 18th, '21, 11:19

Would you mind posting the output of 'df' as that's a very large amount for shared listed by free in your first post.

edit: A 'free -h' at the same time might be good for a comparison under current conditions /edit
I see my C has been in the sea too long - it's gone rusty.
Posts: 50
Joined: Jun 8th, '19, 09:38
Location: Nottinghamshire, UK

Re: [Solved] system has become VERY swappy

Postby jiml8 » Apr 18th, '21, 16:58

I established long ago that free will report the memory assigned to the VMs as shared memory. I consider this to be incorrect, but it is what it is. That is likely the reason "shared" and "cache" show up as so very high.

Anyway, per your request:
Code: Select all
root@dadsbox:~> free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi        11Gi       1.1Gi        17Gi        18Gi       2.2Gi
Swap:         2.0Gi       1.9Gi        70Mi
root@dadsbox:~> df
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                        16G     0   16G   0% /dev
tmpfs                           16G  248M   16G   2% /dev/shm
tmpfs                           16G  2.1M   16G   1% /run
/dev/sdc1                       53G   21G   30G  41% /
tmpfs                           16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/crypt_sda1         2.7T  2.1T  650G  77% /mnt/sda1
/dev/mapper/crypt_sdd5         457G  342G   92G  79% /mnt/sdd5
/dev/mapper/crypt_sdb1         3.6T  2.9T  754G  80% /mnt/sdb1
/dev/mapper/crypt_sdc5         405G  348G   37G  91% /mnt/sdc5
tmpfs                          3.2G   19M  3.2G   1% /run/user/501
tmpfs                          3.2G     0  3.2G   0% /run/user/0   29T  6.1T   22T  22% /mnt/Video
/dev/sdf1                      9.0T  4.7T  4.4T  52% /mnt/NAS

Just for reference, /mnt/NAS is an iscsi share on my NAS and /mnt/Video is an NFS share on the same NAS. The space reported for /mnt/Video includes the space occupied by /mnt/NAS ... just a peculiarity of my installation.

I also have turned off a 33GB swapfile that is located on /dev/sda1, leaving only the 2 GB swap that is on /dev/sdd1/. This is an experiment to see if I can control this obnoxious behavior by restricting swap to an SSD, rather than allowing a spinning disk to be used, You will note that this swap is currently showing as full, but the system is not showing any out of memory + out of swap symptoms. Yet, anyway.
Posts: 1165
Joined: Jul 7th, '13, 18:09

Return to Advanced support

Who is online

Users browsing this forum: No registered users and 1 guest