[SOLVED] (Mozilla) programs freeze during heavy copy

[doktor5000]

Hello,

someone else noticed this problem too, already reported on Mandriva Cooker ML?
http://lists.mandriva.com/cooker/2011-07/msg00036.php
I have that too, whether internal sata drive or external usb drive does not matter.
Worst part is that the system really freezes, sometimes KDE is unusable for 10-25 seconds,
it doesn't react to windows switching or something else.

processor load is normal, something like 15% altogether and under 10% for mount.ntfs.
Mounts also look normal:

Code: Select all

$ mount
[...]
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/sda1 on /media/win_c type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions)
/dev/sda2 on /media/win_d type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions)
/dev/sda3 on /media/win_e type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions)
[...]
/dev/sdb1 on /media/EXTERN3 type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions)
/dev/sdb2 on /media/EXTERN4 type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions)
/dev/sdc1 on /media/EXTERN5 type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions)

Also iotop does not show unusable high I/O.
Don't know where this problem comes from, maybe kernel scheduler changes between 2.6.33/36 to 2.6.38?

[doktor5000]

When not copying with dolphin, but via midnight commander, it's not as bad, only dolphin is still sluggish.
Seems to be that the kio-file process could be the culprit.

[zugunder]

Hi,

I do not have exactly the same problem, but experienced something similar during copying files to NTFS partitions using Krusader.
It kept telling me that it could not save (or modify) file permissions, but copying itself was OK.
I got rid of it by changing the value of parameter WIN_PARTS_UMASK in MSEC to -1 from the default 000.
Try it, maybe it will help.

[doktor5000]

No, that is not the problem, i know that one.
It cant save the permissions, as NTFS doesn't allow to use the unix permissions one-to-one.
But thanks for your suggestion.

[doktor5000]

The problem seems to be the notifications about file transmission progress.
Disabled them, and now the system is way more responsive during copy processes.
Well, but no progress indicator anymore. Starting to not like those KDE regressions with 4.6x ...

Thread can be marked [solved].

[doktor5000]

Sorry, please reopen, the problem is not solved.
After rechecking with bigger copy operations like 5-10 GB at a time,
from one NTFS partition to another, or decompressing multi-GB archives on NTFS partitions,
the problem is still there. ATM i'm trying other I/O schedulers, which can be switched per device.
It's really bad, as the whole Desktop freezes sometimes for half a minute, which makes it
barely usable to unusable during longer copy processes, say for 20-45 minutes.

Not being able to use my box while copying some files around is not really nice, no?

Nobody else has problems with NTFS writing?

[isadora]

[doktor5000]

Thanks isadora

Currently i'm trying out noop scheduler for the external disk which i'm using mostly.
One can find out which scheduler is used for which device (in my case sdb) like this:

Code: Select all

$ cat /sys/block/sdb/queue/scheduler
[noop] deadline cfq

It has gotten better, the system is more responsive and usable again, but not perfect.

You can switch them as root like this:

Code: Select all

echo noop > /sys/block/sdb/queue/scheduler

For more information and a short comparison of the schedulers maybe look here:
http://www.redhat.com/magazine/008jun05 ... chedulers/
http://www.wlug.org.nz/LinuxIoScheduler

[doktor5000]

Actually i'm not sure if this wasn't a file-system problem. After checking the fragmentation for the
partitions in question was way above sane levels, between 25% and ~75% (75% was for a 1TB partition).
After defragmenting those partitions, i'm rechecking, also with cleanly formatted, empty partitions.

[doktor5000]

UPDATE: The cause (and a solution) has been found!

On 12/19/2011 07:26 AM, Franck Bui, new Mandriva kernel maintainer, wrote on mandriva cooker mailing list:

[Cooker] System can stall under heavy IOs with slow storage devices

Hello,

I've noticed a regression on my system which can stall some applications
(most notably firefox) during very very long time (> 10 mins).

It's apparently related to transparent huge pages and I found that it
had already been described by LWN here a month ago:
http://lwn.net/Articles/467328/.

Unlike what it's stated in the article, the regression is not really
hard to trigger in my case: just plug a USB stick, copy on it a big
file, then try to use firefox...

To fix this, simply try to disable THP:

echo madvise >/sys/kernel/mm//transparent_hugepage/enabled

and firefox should work as expected now.

So, if no one objects, the plan is to simply default the THP stuff to
"madvise" for desktop flavour where an application has to ask
explicitly huge page otherwise it will be disabled.

quoted from http://lists.mandriva.com/cooker/2011-12/msg00242.php

[doktor5000]

Shortly after that a kernel update candidate went into validation, check https://bugs.mageia.org/show_bug.cgi?id=3825

[wobo]

Just for the records:
I used this fix on my Mageia 1 (kernel 2.6.38.8-desktop-8.mga) and it works all right, solved a similar problem:
Whenever I transferred an iso file to a usb stick firefox behaved sluggish, almost frozen. This is gone now.

[doktor5000]

@Mods: Thread can be marked as [SOLUTION] or solved, whatever you think fits best, and topic maybe adjusted
to "(Mozilla) programs freeze during heavy copy / I/O operations" or something similar, which is more meaningful and precise than the current one.

[wsams]

" To fix this, simply try to disable THP:

echo madvise >/sys/kernel/mm//transparent_hugepage/enabled

and firefox should work as expected now.
"

Can someone clarify the mechanics of the fix for me? my machine is almost unusable due to the hangup described in this thread

If I open the teminal to a root level prompt and enter the command given above, will this change the /sys/kernel/mm//transparent_hugepage file, or would I have to open it in an editor and make the change?


thank you for your help

Walter

[wobo]

Just enter this command as root, it will do the file editing.

[doktor5000]

For clarification, everything below /sys (and /proc similarly) are no real files, but pseudo-file representations of kernel runtime configurable options.
So if you write another value into that "file" it will just change a kernel option, in this case it will change the transparent huge pages behavior,
so that only those programs who explicitly request huge pages will get them.

If you want to read on about the problem and some background context information about it's mechanics, you can check http://lwn.net/Articles/467328/

[wobo]

Ich habe ja vor einiger Zeit (4.1.12) diesen Fix implementiert und es gab danach keine Verzögerungen mehr beim FF während ich eine ISO auf einen USB-Stick brannte. Das war mit dem Kernel 2.6.38.8-8. Heute habe ich wieder die Verzögerung gehabt. Es gab zwischenzeitlich ein Kernelupdate (2.6.38.8-9). Muss ich eventuell nach jedem Kernelupdate den gleichen Befehl wieder eingeben, da es ja eine Änderung einer Kerneloption ist, die durch eine virtuelle Datei ausgelöst wird. Neuer Kernel, Reboot, virtuelle Dateien neu aufgebaut - ergo: Befehl muss für den neuen Kernel wiederholt werden. Ist das zu laienhaft dargestellt oder sogar falsch gedacht?

Auf jeden Fall ist der Fehler wieder vorhanden, und das in meinem so stabilen Mageia 1!

[doktor5000]

To reply in english with kernels prior to 2.6.38.8-9 you have to do this after every reboot, as it's a runtime configure option.
2.6.38.8-9 has it builtin as default.

[wobo]

Ah, excuse me, I've been reading and writing in the German forums for the last hour.... Of course everybody here understood what I wrote. Thx for not reporting me to the head mistress (1)!

In short:
The issue is back on my machine and I assumed that I had to repeat the command after a new kernel was installed.

2.6.38.8-9 has it builtin as default.

Hmm, that means the problem has another cause on my machine. It came back after I installed the kernel update to 2.6.38.8-9.

(1) for non British: female principal of a school

[doktor5000]

Just installing it doesn't mean you're actually using it, which needs a reboot ... Just to make sure
Have you verified via uname -a that you're actually using that kernel?

[wobo]

Just installing it doesn't mean you're actually using it, which needs a reboot ... Just to make sure
Have you verified via uname -a that you're actually using that kernel?

Yes, masterrrr! I usually obey urpmi's orders.

Code: Select all

[root@marvin ISO]# uname -a
Linux marvin 2.6.38.8-desktop-9.mga #1 SMP Tue Dec 20 09:45:44 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

[doktor5000]

Well, the only other thing i can think of to be really sure is check the value:

Code: Select all

cat /sys/kernel/mm//transparent_hugepage/enabled

If this is set to madvise, then i'd say your current freezes have another reason.

[wobo]

Well, it is set to madvise.
I did a lot of "burning" isos to usb sticks lately on my search for a nice system for my netbook:
- almost freezes until I applied the patch
- not one single freeze after I applied the patch
- almost freezes after kernel update.
What would you think? I was so happy to solve the only issue I have with Mageia 1 - now it's back.

[doktor5000]

Well you could still use the older kernel and apply the fix via /etc/rc.d/rc.local or something similar, no?

[wobo]

Yes, I could certainly do that as a workaround. But "a solution would be better" (a recent advice by somebody here in this forum).
It nags me that this issue turned up after I had a perfect stable installation of a stable release - that's what I call degression.