Page 1 of 1

[Solved] system has become VERY swappy

PostPosted: Feb 12th, '21, 20:51
by jiml8
My last updates seem to have returned my consoles to me, and seem to have taken care of another problem apparently in dbus. So that's good.

But now my system loves my swap. I have turned vm.swappiness down to 1 in order to minimize swapping, but at this time something in excess of 6 GB of my swap is in use. Specifically,
Code: Select all
root@dadsbox:jiml> sysctl -a | grep swapp
vm.swappiness = 1

root@dadsbox:jiml> free
              total        used        free      shared  buff/cache   available
Mem:       32784036     6392636     1513528    19836636    24877872     6049716
Swap:      36909400     6666660    30242740


Other than the update, nothing at all has changed in the system. My normal usage pattern continues, and there should be enough RAM to keep the swap to a minimum. I normally expect the system to be using less than 1GB of swap.

This usage of swap is causing noticeable slowdowns when I access a VM that I have not accessed for a few hours. I do have the swap available, just in case, but this is ridiculous.

Does anyone have any ideas?

Re: system has become VERY swappy

PostPosted: Feb 13th, '21, 03:43
by doktor5000
Did you check what is actually using swap currently, with something like smem or similar ?

Re: system has become VERY swappy

PostPosted: Feb 13th, '21, 18:42
by jiml8
I was not familiar with smem, but that seems like a very useful tool.

The big culprits are chromium, akonadi, and vmware. One of my VMs is the biggest single user of swap. I don't know how that could be, but I am looking into it.

I upgraded to vmware workstation 16 a couple of months ago, but this assault on my swap did not start until after the latest mageia updates that I installed. But perhaps there is some new vmware setting somewhere that is allowing this.

Re: system has become VERY swappy

PostPosted: Feb 17th, '21, 00:31
by jiml8
There is a global setting in vmware workstation to select how swap is used with VMs. You can not allow any VM memory to be swapped, or allow some to be swapped, or allow swapping to occur aggressively.

For many years, I have had my workstation installation set to allow some memory to be swapped, and this has worked well.

No longer.

Now, I have changed the setting to do not allow any VM memory to be swapped, and my swap usage has dropped, while my VM performance has improved.

Something has changed in the most recent kernel, and likely something is also different in Workstation 16 vs Workstation 14 (which was the upgrade I made...14 to 16), and the combination made performance crappy with workstation allowed to swap memory.

Now, memory is getting swapped anyway when it needs to happen, but the host kernel is doing it, not Workstation.

So, I don't know what the details are but the setting change in Workstation has made some big improvements. Swap usage is still much higher than I am used to seeing (right now, about 4.5 GiB in use) but it is well down from what I was seeing before the setting change.

Re: [Solved] system has become VERY swappy

PostPosted: Feb 21st, '21, 00:27
by jiml8
While my configuration change in vmware has restored acceptable operation to the overall system, I am still seeing swap usage creeping up steadily though not rapidly, even as the system's behavior does not seem swappy.

So, while I am not sure there is a problem here, something has definitely changed and I would very much like to know what it is.

I seem to be zeroing in on kcompactd, which is a somewhat new feature that my monitoring tells me is running from time to time, and sucking up a lot of processor when it does run.

Some digging turns up this article:

which describes what is going on.

Again, I am not sure there is a problem here because I do have plenty of swap. Nevertheless, does anyone else here have any comments about this?

Re: [Solved] system has become VERY swappy

PostPosted: Mar 18th, '21, 23:09
by jiml8
I continued to be unhappy with overall system performance, notably in virtual machines.

And I kept seeing kcompactd0 appearing high in my process list, using a lot of CPU.

So, I looked in sysctl, and found this tunable:
Code: Select all

I changed that value to 10, and the difference in system performance is amazing. Much less CPU usage, and the issues I was having with my VMs (particularly the heavily utilized VMs) seems to have vanished.

Re: [Solved] system has become VERY swappy

PostPosted: Mar 19th, '21, 00:13
by morgano
Good find!
Thanks for the update.

Re: [Solved] system has become VERY swappy

PostPosted: Apr 16th, '21, 18:39
by jiml8
This vm.compaction_proactiveness continues to be a problem for me, though it is a greatly reduced problem.

I have adjusted the value down to 1. I just did that; it has been at 5 for the last couple of weeks.

Usually the system is OK, but recently I had an rsync that ran for almost 48 hours, syncing (with hardlinks) between 2 multi-terabyte directories (this was part of a crash recovery where a filesystem was so badly damaged I had to rebuild it).

Given that this rsync had to handle lots of hard links, it wound up occupying a huge amount of RAM, and began impacting my system. Swap usage got as high as 15 GB, and that gawdawful kcompactd0 kept showing up - and every time it showed up, it partly or fully stalled my system until it was either done or gave up. Its attempts to "optimize" my memory usage are making the system hard to use.

I have checked, and neither Mint nor OpenSUSE have this thing compiled in, and I think Mageia would do well to compile it out. It drives swap usage to levels well beyond what is necessary, and in my experience interferes far more than it helps. If I had the option to disable it I would cheerfully do so.

For reference, my system has 32 GB RAM and I for the last few days I have been running 4 VMs along with the host. One is a Mint 19 VM that has 4 GB RAM assigned, one is a Mint 20 VM with 4GB RAM assigned, one is an OpenSUSE Leap 15.2 with 4GB RAM assigned, and one is a FreeBSD 8.4 with 2GB assigned. Thus, I have 14 of my 32GB assigned to VMs. Allow 10% overhead for the VMs, then round up, and half my memory was dedicated to VMs.

The rsync was showing usage of around 3 GB at its peak, and for part of the time I was running bitcoin core, which sucked up another 4 GB or so (I shut it down because I didn't really need it and it was impacting things heavily).

I also was running Firefox with dozens of tabs open and chromium with about 10 tabs open.

Now, chromium is a gross memory hog, but even so there is no way this configuration should have hit 15 GB swap, and that kcompactd0 should not have been crapping all over me.

This configuration is not unlike the configuration I have been running for years. It is even a bit lighter because I did not have any Windows OS's open. I have never had these problems until kcompactd0 appeared. I would be very happy if it disappeared.

Re: [Solved] system has become VERY swappy

PostPosted: Apr 18th, '21, 11:19
by magic
Would you mind posting the output of 'df' as that's a very large amount for shared listed by free in your first post.

edit: A 'free -h' at the same time might be good for a comparison under current conditions /edit

Re: [Solved] system has become VERY swappy

PostPosted: Apr 18th, '21, 16:58
by jiml8
I established long ago that free will report the memory assigned to the VMs as shared memory. I consider this to be incorrect, but it is what it is. That is likely the reason "shared" and "cache" show up as so very high.

Anyway, per your request:
Code: Select all
root@dadsbox:~> free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi        11Gi       1.1Gi        17Gi        18Gi       2.2Gi
Swap:         2.0Gi       1.9Gi        70Mi
root@dadsbox:~> df
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                        16G     0   16G   0% /dev
tmpfs                           16G  248M   16G   2% /dev/shm
tmpfs                           16G  2.1M   16G   1% /run
/dev/sdc1                       53G   21G   30G  41% /
tmpfs                           16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/crypt_sda1         2.7T  2.1T  650G  77% /mnt/sda1
/dev/mapper/crypt_sdd5         457G  342G   92G  79% /mnt/sdd5
/dev/mapper/crypt_sdb1         3.6T  2.9T  754G  80% /mnt/sdb1
/dev/mapper/crypt_sdc5         405G  348G   37G  91% /mnt/sdc5
tmpfs                          3.2G   19M  3.2G   1% /run/user/501
tmpfs                          3.2G     0  3.2G   0% /run/user/0   29T  6.1T   22T  22% /mnt/Video
/dev/sdf1                      9.0T  4.7T  4.4T  52% /mnt/NAS

Just for reference, /mnt/NAS is an iscsi share on my NAS and /mnt/Video is an NFS share on the same NAS. The space reported for /mnt/Video includes the space occupied by /mnt/NAS ... just a peculiarity of my installation.

I also have turned off a 33GB swapfile that is located on /dev/sda1, leaving only the 2 GB swap that is on /dev/sdd1/. This is an experiment to see if I can control this obnoxious behavior by restricting swap to an SSD, rather than allowing a spinning disk to be used, You will note that this swap is currently showing as full, but the system is not showing any out of memory + out of swap symptoms. Yet, anyway.

Re: [Solved] system has become VERY swappy

PostPosted: May 9th, '21, 17:54
by jiml8
Last monday, my workstation experienced a catastrophic hardware failure. This motivated an upgrade.

As of Friday evening, I am running a system with 64GB of RAM, not 32GB. Also, of course, a new processor and motherboard but those are not relevant to this discussion.

My software configuration remains exactly the same, and as I write this I am running the same number of VMs as the last post where I listed my VMs, with the exception that I am running a FreeBSD 12 instance with 4GB assigned rather than the FreeBSD 8.4 with 2 GB, and I have enlarged the memory on the Mint VMs so that each has 8 GB rather than 4 GB, thus relieving some performance constraints that both of those were experiencing.

And look at these free and vmstat commands:
Code: Select all
 jiml@dadsbox:jiml> free
              total        used        free      shared  buff/cache   available
Mem:       65829968    10559752     1208024    26121568    54062192    28420428
Swap:      36909400     2125148    34784252
jiml@dadsbox:jiml> vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 2125148 1244336 15672744 38354208    1    2    83    63   15   13  2  4 92  2  0

This system is showing 28GB available, and yet it is still using 2 GB of swap. It has been up for 33 hours now.

I also have seen that lovely kcompactd0 appearing in the task list, though it isn't staying there for long and isn't hanging this system up at all.

How do I turn that "feature" off completely? It is worthless and is doing far, far more harm than good.

Re: [Solved] system has become VERY swappy

PostPosted: May 10th, '21, 00:57
by doktor5000
jiml8 wrote:How do I turn that "feature" off completely? It is worthless and is doing far, far more harm than good.

Take a look at ... shuge.html ... emory.html ... databases/

Seems you need to at least completely disable transparent hugepages via transparent_hugepage=never kernel option.

Re: [Solved] system has become VERY swappy

PostPosted: May 10th, '21, 02:20
by jiml8
I have been reading up on that, and I don't think disabling hugepages will do it. Seems that hugepages particularly applies to userspace applications getting hugepage allocations, and my system shows that the hugepages allocated is 0. Apparently in a modern system, this is the case most of the time. Also, particularly, userspace memory allocations using hugepages cannot be swapped out.

There was a kernel change late last fall that kicked this off; prior to that, kcompactd0 was not a problem. I think there is some interaction involving all my virtual machines (presently I am reading up on vmware memory "ballooning") but so far I have not found it.

Re: [Solved] system has become VERY swappy

PostPosted: Jul 18th, '21, 02:33
by iamnumpty
I have observed similar behaviour on a laptop with 8GB RAM and Ryzen 7 pro.

Kcompactd0 pegs at 100% and Host(Ubuntu 20.4) itself remains responsive but VM (Windows 10) freezes until kcompactd0 is done with.

I did the default install with LUKS and Ubuntu decided to set SWAP to 1 GB. Initially I thought system wants to swap and it can’t, so I went through all the hassle of reducing rootfs and shrank lv and expanded swap to 16GB. Turned out it was useless.

I then disabled THP, again did not help.
If I disable 3D acceleration the situation is tad better but some of my apps need DirectX so that’s not an option.

The only thing that helped (not permanently) is to keep the VM memory + Overhead less than 50% of System RAM.
If I set VMware preferences to fit all VMs with in 3.5 GB memory, then set VM mem to 3GB and Video memory 128MB, it keeps kcompactd0 at bay for much longer. And when it shows up it goes away far quicker.