Mageia forum

by **Trikki** » Apr 18th, '13, 19:09

I am using the dd command to wrtite iso images to usb stick, so that I can use them on my small Acer laptop. The command has worked fine many times, but lately there have been these weird problems. I have searched around the web but can not seem to find any answers about this. Hope that someone can help with this.

So here is what is happening:

1. I open konsole and su to root
2. I use dd if=image.iso of=/dev/sdc bs=4k
3. Usb stick light starts blinking and cpu usage jumps to 75-100%
4. After a while dd says it has finished after maybe 15-30 minutes depending on the iso size. (Though sometimes this takes way too short time (1-2 min), so I know something is wrong then.) Usb light keeps blinking cpu usage level remains high.
5. I wait for some more time (10-15 minutes) and after nothing happens I use the sync command. Sync how ever does nothing: usb light keeps blinking and cpu usage remains high. And they just keep going until I shut down or yank the usb stick out. The longest time I have waited after dd has finished and after using sync is about an hour.

So any ideas?

I have tried to use top to see if there is a way to kill the ongoing dd command, but there does not apear to be any process with the name dd in it, so I really do not know what to look for.

And also maybe it is worth to mention that I have had one total system freeze once while using the dd command. This has not since repeated.

My main concern is that my hardware is broken...

I have a HP635 Laptop (Dual Core, AMD), with 4G of ram.
I am using Mageia 2 64-bit version, KDE

by **dedanna1029** » Apr 18th, '13, 20:00

Have you tried:

Code: Select all: cd /path/where/iso/is dd if=image.iso of=/dev/sdc (without options)

?

The next question I suppose would be if you run df -h, is that the drive you want to dd to? Mine had to dd to /dev/sdb1. If you dd to the wrong place, it will do some rather quirky things.

The only other thing I can think is maybe the usb stick is bunk. It should also be umounted when you perform the dd.

top you would need to run as root to see dd running.

by **martinw** » Apr 18th, '13, 20:21

Given it has worked fine many times before and has only recently started failing, I would think it is most likely a hardware fault. USB memory sticks don't last forever (and are susceptible to static damage), so the first thing I'd try is a new USB stick. Get one with a shrouded connector, as that reduces the risk of static damage (you're less likely to touch the contacts).

by **Trikki** » Apr 18th, '13, 20:30

dedanna1029 wrote:Have you tried:
Code: Select all
cd /path/where/iso/is dd if=image.iso of=/dev/sdc (without options)
?

The next question I suppose would be if you run df -h, is that the drive you want to dd to? Mine had to dd to /dev/sdb1. If you dd to the wrong place, it will do some rather quirky things.

The only other thing I can think is maybe the usb stick is bunk. It should also be umounted when you perform the dd.

I do not believe that the problem is in in the dd commands I have used, since I have used it numerous times with success. It is clearly writing something to the connected usb stick.

I was merely using the

Code: Select all: dd if=image.iso of=/dev/sdc bs=4k

(Hey I just figured out how to use the code box thing

)

to refer to the command in a general way, so that it would be clear what I was talking about... The bs=4k just tells dd to use 4k bytesize, which altho slower than the default, should be more stable. It could be a factor in this, I'll have to do more tests.

And

Code: Select all: sync

(Is this too much? Should every command be in the codebox?)

is used to flush file system buffers -> description in the man page "Force changed blocks to disk, update the super block.". Basicly the command tolds system to write data from memory buffers to disk, if I have understood it correctly...

I haven't got many usb stick, and only two are "idle" enough to be used for liveUSB tests. Both of these seem to have this same issue. The same thing with my sd-card.

What's even more saddening with this thing is that the problem is not always present (only most of the times), and when problems occur it pretty much makes my computer useles for 30 minutes and more, so it's no small thing to try to test this.

EDIT:
btw, good point on the having to be root to see dd thing. I'll have to test that next time. I can not remember did I log out before using top -> since I was root when running dd and after it stopped if I run top I should have been root then also...but can not remember for sure.
And also worth mentioning: I did run sync as root.

by **Mayavimmer** » Apr 19th, '13, 09:40

Some things to try if you haven't already:

1. tail -f /var/log messages -- you may be able to catch some hardware errors this way.

2. Using the USR1 signal to ask dd for a progress report during copy. Use dd --help to learn how to use it.

3. Use Pipe Viewer, that is the pv command, instead of dd to get a continuous progress report and see if, when and how the copy gets stuck.

4. Use fuser and/or lsof on related processes and devices to understand who is doing what to whom. They can get over detailed if you are not careful.

5. Use the live proc filesystem to get more info on dd's output file descriptor and/or cgroup and go from there.

6. Use Dtrace injection to ... aww, scrap that. Get new stick instead.

by **JoesCat** » Apr 20th, '13, 10:35

The bs=4k is far too small when you are using dd, try using a larger value such as
dd if=image.iso of=/dev/sdc bs=1M

Another thing to note is that some of these smaller netbooks, particularly the ones with econo solid-state drives tend to go bad after some point in time, so you are attempting to dd into some dead sectors, which dd doesn't like doing. In that case, you're better-off doing a fresh install, and let the install process determine which dead-sectors to bypass. Then you can dd this newer image okay for a while before more dead sectors get in the way.

In terms of the USB stick, same issue, use large blocks, like 1M. I've killed a couple of USB sticks with tiny block sizes since you're doing a lot of unnecessarily small block sizes. Solid-state memory such as USB sticks are going to last about 100k writes, and if you do writes of 512bytes into a stick that internally is setup for 16k blocks (actual physical chips, not OS or filetype), then you are basically writing 32x into the same 16k block because you've added 32x 512bytes into that 16k block.
If you want a good technical idea of this read/write stuff, goto the microchip website and lookup eeprom endurance, google for the words microchip eeprom AN537
This will give you a fairly good idea of USB sticks and solid state drives.

by **dedanna1029** » Apr 22nd, '13, 16:28

+1 to both - was trying to get some basics out of the way first. Also, I do know what sync does. With dd, it's not going to do anything insofar as the dd process itself goes. Insofar as a frozen machine, keep in mind Alt+SysRq+REISUB. This puts things into emergency mode first, then syncs. Good practice.

I've done this quite a bit for my netbook, btw, as I got no cd/dvd drive for it, either. Have never had a problem so far.

by **Trikki** » Apr 22nd, '13, 21:13

Thank you all for your help! I am going to call this one solved, atleast for now...

I bought a new usb stick and used bs=1M this time. No problem this time. The cpu remained much lower this time and although dd told it was done before the cpu went down it was back to normal before I even got to send the USR1 signal to it or anytihing.

I believe that the main cause of this was that the too small bytesize had broken both of my live/testUSB-sticks.

EDIT:

When I explained what the commands mentioned do, I ment not to be disrespectful to anyone: My goal was to tell people "This is what I think this and this does..." so that it would be easier to spot if I was going wrong somewhere thinking that I was doing something when in reality was doing something completely difrent... As it seems was the case with the bs=4k -> I was thinking it was a good thing to do, when in reality it was not a very good idea...

by **linuxero** » Apr 24th, '13, 15:06

Just curious; are you dumping the ISO onto the device itself /dev/sdc? Why aren''t you dumping it to a partition?

by **doktor5000** » Apr 24th, '13, 22:51

A bootable ISO is always to be dumped onto the device itself, as it contains partitions.

by **linuxero** » Apr 25th, '13, 09:59

doktor5000 wrote:A bootable ISO is always to be dumped onto the device itself, as it contains partitions.

I have a 2GB memory card, microSD, on which I have created several partitions for PartedMagic, SystemRescueCD, XUbuntu and Mageia Live. For Mageia I dumped it using dd as I couldn't manage to make it bootable any other way then I am using grub to boot the one I want..does that make me doing something wrong then? - as far as Mageia ISO is concerned..

by **doktor5000** » Apr 26th, '13, 18:43

Well, a multiboot setup on one drive is not the standard use case

Booting several images via a bootloader is currently the only way to do this AFAIK. Although you could take a look at YUMI:
http://www.pendrivelinux.com/yumi-multi ... b-creator/

by **Trikki** » Apr 27th, '13, 20:33

Maybe I was too quick to call this one solved: The secon time I used my new USB-stick with dd I once again had a complete system freeze. Trying to log into a console with Ctrl+Alt+Fn just made the screen go black and nothing seemed to happen. I managed to get out of it by pressing Ctrl+Alt+Backspace (= restart X) a few times.

After this I went thru the logs once again... In /var/log messages there are a dizzying amount of stuff in the two seconds after dd has started and I really do not know what to look for. It is probably pointless to drop the whole mess of output here? I would rather be interested to know just what excactly am I supposed to look for?. (RANT WARNING! This, as a side note, has frustrated me for a long while: It is always said: "Read the log files, read the log files..." Well I have tryied, but there is so much information there! And all of it is quite vague: "axpcid sighup exit signal b14x0026621e. Unexpected forearch: Dbus pid 001.001.884461 Not found" etc. [<- Not a real example from actual log, I'm just trying to make a point here.] If anyone has any info about a book or a website that has some instructions on how to actually understand that stuff I would love to know. I have tried to find, but perhaps not have found the right search words. I have even tryed to look at programs such as Logwatch that send important log file entries to sysadmins email, but have not found out just what exacly those programs consider to be "important" log entrys...END OF RANTING.]

The only clearly interesting entry I found from that time period was:

Code: Select all: Out of memory: Kill process 2149 (plasma-desktop) score 9 or sacrifice child

This was in the logs for a couple of times. Could this mean that dd makes my plasma-desktop crash?

by **Trikki** » Apr 27th, '13, 20:38

linuxero wrote:Just curious; are you dumping the ISO onto the device itself /dev/sdc? Why aren''t you dumping it to a partition?

Yes. In every source that I have read it is always sayd that you absolutely must use the device itself, sdc NOT sdc1 etc. So that is why. As to why it must be done that way, this I have never bothered to find out... Maybe I should, but my list of thing to find out is getting to be way too long, so I'm not sure when I get to it

by **Mayavimmer** » Apr 27th, '13, 21:12

Trikki, let me see if I understand you: you say that since the logs contain a "dizzying" amount of stuff and it's "probably pointless to drop the whole mess of output here" then you are not going to give us ANY diagnostic message, not even ONE line, except for a line of gibberish, admittedly "not a real example from actual log". I tried, but I can't figure you out.

Try the following procedure; it's very easy:

1. Open 2 Konsole terminals.
2. In one you run "tail -f /var/log/messages"
3. Watch the output until you find a quiet break and the messages stop coming fast and furious.
4. Press ENTER a few times to create some white space before the dizzying stuff begins.
5. Run your dd command in the other terminal.
6. Switch quickly back to the first terminal and interrupt the tail command with control-C before dizzying messages fill up the Konsole buffer.
7. Pick a good number, say 20.
8. Copypaste the first 20 lines of precious log messages we need to help you with, starting right after the white space.

Btw, your rant also seems wrongheaded to me. What you are asking for amounts to finding a universal troubleshooting procedure good for every problem. Impossible. The best you can do is to start with the error messages and then go from there.

Patiently waiting for the blasted info...

by **doktor5000** » Apr 27th, '13, 22:45

If your system kills processes due to OOM (out of memory) when using dd, then you can try to use some flags for dd.
That is what i do at work when writing bigger images (>10GB) to flash drives, as otherwise dd tries to read as much of the source image
file into free system RAM, which really gives huge performance hits.
Use dd as you would normally, but append

Code: Select all: oflag=direct iflag=direct

by **Trikki** » Apr 28th, '13, 12:14

Mayavimmer wrote:Trikki, let me see if I understand you: you say that since the logs contain a "dizzying" amount of stuff and it's "probably pointless to drop the whole mess of output here" then you are not going to give us ANY diagnostic message, not even ONE line, except for a line of gibberish, admittedly "not a real example from actual log". I tried, but I can't figure you out.

Using words is not my strongest skill. I am very sorry for my bad choice of words. I was quite frustrated at the moment of writing the previous post. I think you understood me wrong, but it's all my fault for not speking more clearly. So trying here to be more clear:

1. In the var/log/messages there are about 300+ lines in the couple of seconds after dd was started. What I was trying to ask was: Is it usefull for me to drop everything in here, or would it be more usefull if I tryed to look for something more specific and then report that? I was trying to spare you from the trouble of having to go thru all of that information. I was also hoping that I could get to learn more about analysing logs rather than just droping the whole thing here and asking someone else to do it.

But here is some of it then, I tryied to remove the duplicates to make it more readable, as these same messages were written it the log for multiple times:

Code: Select all: Apr 27 20:21:35 localhost kernel: [ 3952.382219] dd invoked oom-killer: gfp_mask=0x10200d0, order=0, oom_adj=0, oom_score_adj=0 Apr 27 20:21:35 localhost kernel: [ 3952.382230] dd cpuset=/ mems_allowed=0 Apr 27 20:21:35 localhost kernel: [ 3952.382236] Pid: 5375, comm: dd Tainted: G C O 3.4.34-desktop-1.mga2 #1 Apr 27 20:21:35 localhost kernel: [ 3952.382240] Call Trace: Apr 27 20:21:35 localhost kernel: [ 3952.382254] [<ffffffff810bfce1>] ? cpuset_print_task_mems_allowed+0x91/0xa0 Apr 27 20:21:35 localhost kernel: [ 3952.382262] [<ffffffff8110c356>] dump_header.isra.10+0x76/0x1a0 Apr 27 20:21:35 localhost kernel: [ 3952.382270] [<ffffffff8123fe60>] ? ___ratelimit+0xa0/0x120 Apr 27 20:21:35 localhost kernel: [ 3952.382276] [<ffffffff8110c6f5>] oom_kill_process.part.12.constprop.13+0x135/0x270 Apr 27 20:21:35 localhost kernel: [ 3952.382283] [<ffffffff811f88f5>] ? security_capable_noaudit+0x15/0x20 Apr 27 20:21:35 localhost kernel: [ 3952.382291] [<ffffffff8105bb97>] ? has_capability_noaudit+0x17/0x20 Apr 27 20:21:35 localhost kernel: [ 3952.382296] [<ffffffff8110cd07>] out_of_memory+0x367/0x540 Apr 27 20:21:35 localhost kernel: [ 3952.382303] [<ffffffff81112316>] __alloc_pages_nodemask+0x916/0x930 Apr 27 20:21:35 localhost kernel: [ 3952.382311] [<ffffffff8118fe72>] ? alloc_buffer_head+0x22/0x50 Apr 27 20:21:35 localhost kernel: [ 3952.382318] [<ffffffff81148426>] alloc_pages_current+0xb6/0x120 Apr 27 20:21:35 localhost kernel: [ 3952.382323] [<ffffffff8110881f>] __page_cache_alloc+0xcf/0xf0 Apr 27 20:21:35 localhost kernel: [ 3952.382328] [<ffffffff811095e9>] grab_cache_page_write_begin+0x79/0xf0 Apr 27 20:21:35 localhost kernel: [ 3952.382334] [<ffffffff811971e0>] ? blkdev_get_blocks+0xd0/0xd0 Apr 27 20:21:35 localhost kernel: [ 3952.382339] [<ffffffff81192d98>] block_write_begin+0x38/0xa0 Apr 27 20:21:35 localhost kernel: [ 3952.382343] [<ffffffff8110865a>] ? unlock_page+0x2a/0x40 Apr 27 20:21:35 localhost kernel: [ 3952.382348] [<ffffffff81196783>] blkdev_write_begin+0x23/0x30 Apr 27 20:21:35 localhost kernel: [ 3952.382353] [<ffffffff81108bc2>] generic_file_buffered_write+0x112/0x290 Apr 27 20:21:35 localhost kernel: [ 3952.382359] [<ffffffff8110a291>] __generic_file_aio_write+0x231/0x440 Apr 27 20:21:35 localhost kernel: [ 3952.382364] [<ffffffff81196f33>] blkdev_aio_write+0x33/0x90 Apr 27 20:21:35 localhost kernel: [ 3952.382370] [<ffffffff811608ba>] do_sync_write+0xda/0x120 Apr 27 20:21:35 localhost kernel: [ 3952.382375] [<ffffffff81077dc5>] ? __srcu_read_lock+0x25/0x30 Apr 27 20:21:35 localhost kernel: [ 3952.382382] [<ffffffff8108795d>] ? set_next_entity+0x9d/0xb0 Apr 27 20:21:35 localhost kernel: [ 3952.382388] [<ffffffff811f90ac>] ? security_file_permission+0x2c/0xb0 Apr 27 20:21:35 localhost kernel: [ 3952.382394] [<ffffffff81160e51>] ? rw_verify_area+0x61/0xf0 Apr 27 20:21:35 localhost kernel: [ 3952.382399] [<ffffffff811611b3>] vfs_write+0xb3/0x180 Apr 27 20:21:35 localhost kernel: [ 3952.382403] [<ffffffff811614da>] sys_write+0x4a/0x90 Apr 27 20:21:35 localhost kernel: [ 3952.382410] [<ffffffff814672bd>] system_call_fastpath+0x1a/0x1f Apr 27 20:21:35 localhost kernel: [ 3952.382413] Mem-Info: Apr 27 20:21:35 localhost kernel: [ 3952.382416] Node 0 DMA per-cpu: Apr 27 20:21:35 localhost kernel: [ 3952.382421] CPU 0: hi: 0, btch: 1 usd: 0 Apr 27 20:21:35 localhost kernel: [ 3952.382425] CPU 1: hi: 0, btch: 1 usd: 0 Apr 27 20:21:35 localhost kernel: [ 3952.382428] Node 0 DMA32 per-cpu: Apr 27 20:21:35 localhost kernel: [ 3952.382432] CPU 0: hi: 186, btch: 31 usd: 2 Apr 27 20:21:35 localhost kernel: [ 3952.382435] CPU 1: hi: 186, btch: 31 usd: 0 Apr 27 20:21:35 localhost kernel: [ 3952.382438] Node 0 Normal per-cpu: Apr 27 20:21:35 localhost kernel: [ 3952.382442] CPU 0: hi: 42, btch: 7 usd: 29 Apr 27 20:21:35 localhost kernel: [ 3952.382446] CPU 1: hi: 42, btch: 7 usd: 0 Apr 27 20:21:35 localhost kernel: [ 3952.382454] active_anon:89269 inactive_anon:18166 isolated_anon:0 Apr 27 20:21:35 localhost kernel: [ 3952.382456] active_file:331799 inactive_file:401339 isolated_file:2 Apr 27 20:21:35 localhost kernel: [ 3952.382458] unevictable:0 dirty:396295 writeback:5042 unstable:0 Apr 27 20:21:35 localhost kernel: [ 3952.382459] free:20607 slab_reclaimable:24545 slab_unreclaimable:6337 Apr 27 20:21:35 localhost kernel: [ 3952.382461] mapped:16263 shmem:2826 pagetables:6623 bounce:0 Apr 27 20:21:35 localhost kernel: [ 3952.382465] Node 0 DMA free:14708kB min:284kB low:352kB high:424kB active_anon:0kB inactive_anon:0kB active_file:440kB inactive_file:696kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15656kB mlocked:0kB dirty:696kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:60kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2085 all_unreclaimable? yes Apr 27 20:21:35 localhost kernel: [ 3952.382482] lowmem_reserve[]: 0 3499 3609 3609 Apr 27 20:21:35 localhost kernel: [ 3952.382488] Node 0 DMA32 free:65680kB min:65240kB low:81548kB high:97860kB active_anon:356060kB inactive_anon:71496kB active_file:1322112kB inactive_file:1596044kB unevictable:0kB isolated(anon):0kB isolated(file):8kB present:3583528kB mlocked:0kB dirty:1576464kB writeback:19564kB mapped:62352kB shmem:11268kB slab_reclaimable:90924kB slab_unreclaimable:18744kB kernel_stack:2056kB pagetables:25564kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5099394 all_unreclaimable? yes Apr 27 20:21:35 localhost kernel: [ 3952.382506] lowmem_reserve[]: 0 0 110 110 Apr 27 20:21:35 localhost kernel: [ 3952.382511] Node 0 Normal free:2040kB min:2052kB low:2564kB high:3076kB active_anon:1016kB inactive_anon:1168kB active_file:4644kB inactive_file:8616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:112896kB mlocked:0kB dirty:8020kB writeback:604kB mapped:2700kB shmem:36kB slab_reclaimable:7196kB slab_unreclaimable:6596kB kernel_stack:384kB pagetables:928kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:25351 all_unreclaimable? yes Apr 27 20:21:35 localhost kernel: [ 3952.382527] lowmem_reserve[]: 0 0 0 0 Apr 27 20:21:35 localhost kernel: [ 3952.382533] Node 0 DMA: 3*4kB 3*8kB 1*16kB 2*32kB 4*64kB 2*128kB 3*256kB 4*512kB 1*1024kB 3*2048kB 1*4096kB = 14708kB Apr 27 20:21:35 localhost kernel: [ 3952.382547] Node 0 DMA32: 10638*4kB 2123*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 65680kB Apr 27 20:21:35 localhost kernel: [ 3952.382561] Node 0 Normal: 278*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2040kB Apr 27 20:21:35 localhost kernel: [ 3952.382574] 751417 total pagecache pages Apr 27 20:21:35 localhost kernel: [ 3952.382577] 15430 pages in swap cache Apr 27 20:21:35 localhost kernel: [ 3952.382580] Swap cache stats: add 16336, delete 906, find 408/409 Apr 27 20:21:35 localhost kernel: [ 3952.382583] Free swap = 4022268kB Apr 27 20:21:35 localhost kernel: [ 3952.382585] Total swap = 4087272kB Apr 27 20:21:35 localhost kernel: [ 3952.412350] 946160 pages RAM Apr 27 20:21:35 localhost kernel: [ 3952.412358] 34927 pages reserved Apr 27 20:21:35 localhost kernel: [ 3952.412363] 828490 pages shared Apr 27 20:21:35 localhost kernel: [ 3952.412368] 182036 pages non-shared Apr 27 20:21:35 localhost kernel: [ 3952.412919] Out of memory: Kill process 2149 (plasma-desktop) score 9 or sacrifice child Apr 27 20:21:35 localhost kernel: [ 3952.412927] Killed process 2154 (ksysguardd) total-vm:9256kB, anon-rss:524kB, file-rss:768kB Apr 27 20:21:35 localhost kernel: [ 3952.451838] X invoked oom-killer: gfp_mask=0x280d0, order=0, oom_adj=0, oom_score_adj=0 Apr 27 20:21:35 localhost kernel: [ 3952.451848] X cpuset=/ mems_allowed=0 Apr 27 20:21:35 localhost kernel: [ 3952.451854] Pid: 812, comm: X Tainted: G C O 3.4.34-desktop-1.mga2 #1

2. I find it quite strange that you tell me, that "you are not going to give us ANY diagnostic message, not even ONE line, except for a line of gibberish, admittedly "not a real example from actual log"" as I was trying to do just the thing you told me to do: "The best you can do is to start with the error messages and then go from there." I indeed posted the only clear error message from the output aroud the timeperiod:

Code: Select all: Out of memory: Kill process 2149 (plasma-desktop) score 9 or sacrifice child

I was trying to narrow it down.

3. As for my stupid little rant: I tryed to make it obvious "Here be ranting!", so that you could just ignore it as sidetracked ranting. But once again: I am verry sorry, the whole ranting was stupid and childish of me.

4. About logs in general: what I was trying to say: "Log messaged are rather vague by them selves (and I gave a stupid little example just to make my point). Does anybody know of any source (book or website) where one could start studying and learning to understand their meanings in general?" I did not mean that I wanted you to hand me over the dictionary that has answers to every question. As to your point about trying to find some error mesages and start from there, that is just what I have tryed to do so far, but would wish to find out more.

by **Trikki** » Apr 28th, '13, 13:19

doktor5000 wrote:If your system kills processes due to OOM (out of memory) when using dd, then you can try to use some flags for dd.
That is what i do at work when writing bigger images (>10GB) to flash drives, as otherwise dd tries to read as much of the source image
file into free system RAM, which really gives huge performance hits.
Use dd as you would normally, but append
Code: Select all
oflag=direct iflag=direct

This could very likely be the case: there are some amount of "oom" messages in the log. And it seems that dd starts some kind of "oom-killer":

Code: Select all: Apr 27 20:21:35 localhost kernel: [ 3952.382219] dd invoked oom-killer: gfp_mask=0x10200d0, order=0, oom_adj=0, oom_score_adj=0

?

I will most definately try using the

Code: Select all: oflag=direct iflag=direct

next time.

The iso I was dd:ing was only 2.4GB so it should not be all that large as such for my 4GB of ram, should it? Could it still be the cause of the out of memory? I am aware that dd is resourse heavy so I never run any other programs while dd is running, so maximum amount of memory should be free at the moment. Could this problem be a sign of broken ram chips or something like that?

by **doktor5000** » Apr 28th, '13, 13:49

If RAM would be defective, would probably show otherwise. But it's never a bad idea to download and complete one run of memtest: http://www.memtest.org/#downiso

If you want to take a look at how much free memory you have, take a look at free -m (-m for megabytes, easier readable). Here's an example of my system,
running not much more than firefox with ~20 tabs and transmission with ~20 torrents or so. Still, only 1 GB free out of 4, and no swap:

Code: Select all: [doktor5000@Mageia2 ~]$ free -m total used free shared buffers cached Mem: 3923 2836 1086 0 1174 450 -/+ buffers/cache: 1211 2712 Swap: 0 0 0

If i'd use dd, and the other memory cannot be reclaimed, the system will come to a crawl, and tries to free up memory. If it can't (in your log excerpts check for: memory pages reclaimable), and there's no swap available, it will try to kill the processes startest last and using the most RAM. Check f.ex: http://serverfault.com/questions/387731 ... i-avoid-it

by **Mayavimmer** » Apr 28th, '13, 14:39

Trikki, thanks for the useful logs. As a general rule, and probably the only useful thing one can learn about understanding log files, is that usually errors produce avalanches of other errors, therefore only the first few lines are meaningful. In this case only the first line is meaningful, that is the message about dd calling oom_killer.

I don't know why this should happen when you have 4GB RAM and only 2.4GB image written at 1MB blocksize. I would guess some kind of hardware problem. Check the RAM. Try different USB ports -- one shorted out on me a while back. Try a different (live) OS. Try different stick.

One more thing would be moderately useful: I'd like to know the time elapsed between the dd invocation and the first error message. Use the logger command like:

Code: Select all: logger STARTING && dd ...

...so you can compare times in the /var/log/messages output.

About logs and "learning to understand their meanings in general" I have to insist that there is no such thing. The reason is that logs are unsystematic dumps of any useful bits of information the program happens to have on hand at the moment of crash. No grand systematic procedure can be followed by a bit of code that is falling into the abyss... Logs may be produced by thousands of different softwares written by thousands of different programmers under billions of different sets of circumstances. No "generic" ways to understand error messages is possible. It's like random panicky people calling for help on the radio: no book or website will help you understand the contents of their communications.

by **doktor5000** » Apr 28th, '13, 15:50

Mayavimmer wrote:About logs and "learning to understand their meanings in general" I have to insist that there is no such thing. The reason is that logs are unsystematic dumps of any useful bits of information the program happens to have on hand at the moment of crash. No grand systematic procedure can be followed by a bit of code that is falling into the abyss... Logs may be produced by thousands of different softwares written by thousands of different programmers under billions of different sets of circumstances. No "generic" ways to understand error messages is possible.

Absolutely true. One should also understand is that /var/log/messages or /var/log/syslog (or nowadays systemd's journal, queriable via journalctl) are only containers, into which the dozens available log implementations write, on behalf of dozens or hundred client applications: the kernel, all sorts of daemons/services, end-user programs, abstraction layers in between and whatnot.

Nevertheless, after a while of teaching yourself how to read logs, filter out the relevant parts and google the problem context by which this log message was triggered, you get a good feeling and much better (up to >95%) hit chance which log messages are relevant and which are absolutely bogus for the problem at hand, even if they really might look frightening and serious.

Only way to achieve this is learning by doing - and maybe asking others about log excerpts in the process.
That's why its essential to post log entries and debug informations, and others will tell you what part is relevant and what it means.
Problem is that most people are only interested in easy solutions, and don't want to understand the problem behind it.
Next time the same problem is encountered, they will stumble again about that problem and probably re-ask.

by **Trikki** » Apr 28th, '13, 19:15

Mayavimmer wrote:Trikki, thanks for the useful logs. As a general rule, and probably the only useful thing one can learn about understanding log files, is that usually errors produce avalanches of other errors, therefore only the first few lines are meaningful. In this case only the first line is meaningful, that is the message about dd calling oom_killer.

I don't know why this should happen when you have 4GB RAM and only 2.4GB image written at 1MB blocksize. I would guess some kind of hardware problem. Check the RAM. Try different USB ports -- one shorted out on me a while back. Try a different (live) OS. Try different stick.

One more thing would be moderately useful: I'd like to know the time elapsed between the dd invocation and the first error message. Use the logger command like:
Code: Select all
logger STARTING && dd ...

...so you can compare times in the /var/log/messages output.

Thank you for your help!

Just a quick reply here:

The whole thing happened within 5 minutes. From what I can undestand from the logs the

Code: Select all: Apr 27 20:21:35 localhost kernel: [ 3952.382219] dd invoked oom-killer: gfp_mask=0x10200d0, order=0, oom_adj=0, oom_score_adj=0

was written to the logs only a couple of second after dd was started.

Here is the message from when I plugged in the usb-stick:

Code: Select all: Apr 27 20:20:53 localhost kernel: [ 3909.681864] usb 2-1: new high-speed USB device number 2 using ehci_hcd

As can be seen there was only 42 seconds between the time that the kernel recognized the usb stick and the oom-killer message. I was logged in as root and had the dd command ready, plugged in the usb-stick waited a few seconds (waited for the kde notification of a new usb device) and hit enter - and then only seconds later the whole system had completely frozen.

Hope this info is usefull. I'll study it some more tomorrow and report back with more information.

And also, almoust forgot: I run a memtest on my machine. No errors reported there.

by **Mayavimmer** » Apr 28th, '13, 21:37

Just to be on the safe side, run the df command just before running dd to make sure your device has not been mounted by an overeager daemon. Heh heh. All sorts of unpredictable though entertaining things might ensue. Also make sure you wait _every_time_ for the kernel to assign a device name (you have reported /dev/sdc so far) because names can change unpredictably and at the worst possible time, according to old Murphy, even if they have always consistently been /dev/sdc so far. Happened to me. Yes. (One possible reason is that some devices are slow to start at boot and may be ignored by poor systemd programming. Yes.)

by **linuxero** » Apr 29th, '13, 14:20

@doktor5000; thanks but I'd rather use Linux-only-based solutions

@Trikki;

You got me really lost in your problem so starting fresh I'd suggest you do the following:

1. Open a new Terminal
2.

Code: Select all: su -

in there as root
3. use the dd command you want making sure you are using it against the right USB device (dmesg, fdisk...etc). Also make sure that the ISO is not corrupt and readable.
4. Open a second Terminal, be root there too using

Code: Select all: su -

, and find out the PID of dd using pgrep or ps
5. Use the following command to see the progress of dd.

Code: Select all: watch -n 5 kill -USR1 0000

***** Of course change 0000 above with the PID of the dd command you got before.

Now report back the problem in a 1, 2, 3... order, please..

by **Trikki** » Apr 29th, '13, 17:52

linuxero wrote:@Trikki;

You got me really lost in your problem

I am going to do a couple of more thorough tests on this, but unfortunately have no time until maybe wednesday. I'll need about three hours to run the tests, gather information and write a "readable" answer with my broken english. Please be patient.

But in the meantime let me sum up what has happened so far:

1. As can be read from the first few posts I told about dd acting strange. I have used dd for about 80 times in a year timeperiod. The problems have shown up only within the last 20 or so times. It was pointed out that the bytesize I used was way too small and that that might have been the problem. I bought a new usb stick, used bigger bytesize and everything seemed ok. After that I was ready to call this [SOLVED].

2. ...until the next time I used dd --> within seconds: a complete system freeze/crash! So I came back here speculating that maybe this thing was not solved after all... During the freeze/crash I could not open a new terminal to send a USR1 signal or anything like that. I could not even get to a virtual console with Ctrl+Alt+Fn -> all i got from this was a black screen with a small blinking cursor at the left upper corner of the screen. Managed to get out of this by restarting X with Ctrl+Alt+Backspace.

3. So far we have concluded that dd has caused my system memory to overload (Out of memory) which was the cause of the crashing. As to why exactly dd causes my system to run out of memory, that remains a mystery. (Now I have added "?" after the [SOLVED], as situation is unclear at the moment.)

I wil run the mentioned and sugested tests as soon as I have some time and report back.

---
Oh and just one more thing...

Trikki wrote:About logs in general: ... ... Does anybody know of any source (book or website) where one could start studying and learning to understand their meanings in general?" I did not mean that I wanted you to hand me over the dictionary that has answers to every question. As to your point about trying to find some error mesages and start from there, that is just what I have tryed to do so far, but would wish to find out more.

I found out about a book on system logs in general: Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management / Anton A. Chuvakin, Kevin J. Schmidt & Christopher Phillips (ISBN 978-1597496353).

Might not be perfect but atleast its something to starts with...

Mageia forum

[DONE] Strange trouble while using dd (and sync)

[DONE] Strange trouble while using dd (and sync)

Re: Strange trouble while using dd (and sync)

Re: Strange trouble while using dd (and sync)

Re: Strange trouble while using dd (and sync)

Re: Strange trouble while using dd (and sync)

Re: Strange trouble while using dd (and sync)

Re: Strange trouble while using dd (and sync)

Re: Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED] Strange trouble while using dd (and sync)

Re: [SOLVED?] Strange trouble while using dd (and sync)

Who is online