[SOLVED] Advice RE: 4Tb external drive

[Myles]

Hi all, I just bought a new 4Tb drive and installed it in an enclosure to replace my WD MyBook 3Tb used for backups. When I had powered it up and when it was plugged into Mageia the system didn't recognise that it was there, or ask me what to do with it. I checked in MCC and there was no record of it in there. So I booted into my W7 system, into Disk management. Windows asked me to initialise it before it could manage it and asked if I wanted MBR or GPT? I - not thinking too clearly as this is my first drive > 3Tb - I stupidly chose MBR! So I ended up with 1 x 2Tb partition and 1 x 1.6Tb partition. Now even though that was done, when I plug it into Mageia the system still does not acknowledge that it is there, but Windows finds it successfully. So any ideas on why this is, or should I be doing something specific?

Now stupid questions.
1. If I convert the drive to GPT in Windows, will I still have the 2 separate partitions? (easily solved if I do, delete and re-partition to single 4Tb)
2. If I convert the drive to GPT in Windows with either 2 partitions or 1 x 4Tb, will Mageia mount it successfully?

Another advice on using GPT drives most welcome as I haven't used them before.

[magfan]

Which Mageia version do you use? In Mageia 3 mcc uses fdisk which cannot handle huge partitions. So you have to install gdisk first. Once gdisk is installed there should be no problem with recognizing your external disk and creating partitions on it. Just last week I plugged in a new 4TB HGST Touro drive and everything worked fine.

An other possibility is to install gparted and create a single partition on your external disk. Just take care when you save the new partition table: default is "dos" but in your case it should be "gpt". Luckily the program asks for it...

[Myles]

Thanks for the reply.
I am currently running Mageia #3 64bit but I am about to upgrade to #4. I read a post in the forums - after searching and before I submitted this - about fdisk not handling big disks. I have just checked and I already have gdisk 64bit installed on my system. So why doesn't it pick up that the drive is connected? I am connecting it via USB3, which is what I have my MyBook 3Tb connected by.

[artificeprime]

Hi Myles. This is kind of déjà vu. Do you recall last June when you were having some issues with Amarok and the thread you started digressed onto the subject of your (then) new 3TiB external? I replied with what I hoped would be some helpful links.

[Myles]

Thanks for the replies magfan and artificeprime. What I posted about last June was a changed action in Amarok, which turned out to be the way USB drives are mounted in Mageia #3 compared to Mageia #2, not with Amarok itself. I never had a problem per se with actually using my MyBook 3Tb drive - it always worked from when I plugged it in IIRC. I also went to Rod Smiths site (the link you provided) and read all he had posted and was so impressed I donated money to him.
Anyway, I have worked this out myself and am now using my 4Tb GPT drive with a almost 4Tb partition.
Thanks for the help.

[artificeprime]

Glad I could help. Oh, and just a heads-up in case you haven't upgraded to Mageia 4 yet: its MCC still can't handle GPT disks.

[Myles]

artificeprime, thanks. I just just upgraded to Mageia #4. MCC can see my 4Tb GPT disk, with almost 4Tb partition, just fine! What is the issue?

[artificeprime]

Take a look at how MCC's diskdrake displays sda and sdc, the two GPT drives connected to the machine in my signature (sdb is a 200GB conventional MBR drive).

My system drive sda displays correctly, both in its total usable (931GB) and individual partition volume(s). It's a 1TB Seagate SATA internal. Mageia sits on the first four partitions totalling 10GB. I'm no fan of crazy-large partitions, so I slice-and-dice my disks pretty heavily and grow into the unformatted partitions as my needs call for it.

diskdrake_m4_1.png (19.57 KiB) Viewed 3625 times

In order to make sda usable (and bootable) I had to pre-partition it with gdisk off a Mageia CD booted into recovery mode before attempting the install. Since diskdrake is limited to fdisk, its attempts to partition this unit failed.

The trouble becomes abjectly clear looking at sdc. This is a 3TB Seagate GoFlex external connected via USB 2.0. I'm guessing the strange name diskdrake displays is coming off its USB interface or something; I think its actual name (similar to sda) should be ATA PT3000DM.... The total usable volume displays correctly as 2.7TB, but its eleven partitions, each ~250GB, get pancaked off to the left so it appears as if it still has unallocated space. It doesn't, but if I click on this space, diskdrake reports 2.3TB remains empty and would attempt to partition it if I told it to. Needless to say, that would destroy the existing GPT scheme and I would lose all my backed-up data.

diskdrake_m4_2.png (16.83 KiB) Viewed 3625 times

If I select sdc1, diskdrake shows it to be only 31GB in size, not the ~250GB it actually is.

diskdrake_m4_3.png (18.78 KiB) Viewed 3625 times

I was sure we were on the same page regarding this issue and that you saw the same thing as me last June...

...one showing as 394Gb on my 3Tb external hardrive - for some reason unknown to me I have no idea why it only shows as 394Gb in MCC!

I'd also be very surprised if Mageia 4's current MCC/diskdrake doesn't show you the same erroneous output for both your 3 and 4TB drives as I get for my sdc. For the time being, whatever you do, don't modify these drives with MCC/diskdrake.

[Myles]

artiificeprime, my apologies, I didn't read down that far in what I wrote I thought I had only mentioned the changed actions with (what I thought) was with Amarok.
Whoa, thanks for the reminder! Just re-checked my 3Tb under MCC (obviously wasn't paying as close attention as I should have been when I checked disks in MCC earlier!), and it still shows only the 349Gb partition so I never did do anything about it - it must still be MBR, not GPT. I changed my 4Tb from MBR to GPT under Windows and MCC shows the whole 4Tb partition for that. But I won't modifiy/manage either of these under MCC.

On a side note, how did you get those images embedded into your reply. I wanted to in this post (and in previous posts) but didn't know how. I have previously had to upload to a photo site and include the URL in between the "" tags.

[doktor5000]

On a side note, how did you get those images embedded into your reply. I wanted to in this post (and in previous posts) but didn't know how. I have previously had to upload to a photo site and include the URL in between the "" tags.

Below the forum editor (only available in Full editor, not via quick reply), switch to the tab "Upload attachment".
Upload your picture, then when finished, click in the editor window where you want your picture to display,
and click the button "Place inline".

[Myles]

Thanks for that doktor, most helpful. I hadn't noticed that tab before.
Now I had better upgrade my 3Tb to GPT as well.
I will also mark this as SOLVED.
As usual, thanks for all the help from everyone.

[artificeprime]

...so I never did do anything about it - it must still be MBR, not GPT. I changed my 4Tb from MBR to GPT under Windows and MCC shows the whole 4Tb partition for that.

Hold on now. DON'T TRUST ANYTHING MCC SAYS ABOUT THAT DRIVE!

What is the actual size of its partition? Once you mount it, pretty much any source except MCC, diskdrake, and fdisk will give you a reliable answer. What does this report?

Code: Select all

$ df -h /run/media/myles/[partition name]/

If the size comes back as >2TB then your drive is already GPT-scheme partitioned. By definition it has to be; MBR can't manage partitions that large (unless the drive happens to support 4096b physical sectors instead of the conventional 512b).

The way to be 100% sure is to run the following as root:

Code: Select all

# gdisk -l /dev/sdc | head -13

The gdisk flag is a lower-case "L", and remember to substitute your drive's correct device name for sdc

If gdisk replies with results like I get off my 3TB sdc:

Code: Select all

GPT fdisk (gdisk) version 0.8.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sdc: 732566645 sectors, 2.7 TiB
Logical sector size: 4096 bytes
Disk identifier (GUID): 8DA249A3-80FA-4049-ACA1-E4EDD7227DB7
Partition table holds up to 128 entries

...then you're good as you are. MCC's output is simply erroneous. Ignore it.

If, on the other hand, gdisk spits out results like I get for my MBR-scheme partitioned sdb

Code: Select all

GPT fdisk (gdisk) version 0.8.8

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.
***************************************************************

You'll want to go ahead with your plans to repartition.

Incidentally, if you run gdisk in interactive mode (by removing the -l flag and the pipe to head), the message between the asterisks turns into this dire warning:

Code: Select all

***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************

I hope I got this reply out to you in time to save you from any unnecessary effort.

[Myles]

Thanks for that artificeprime, I hadn't done anything as yet!
So I ran gdisk on the 3Tb drive it's MBR!

[root@Bilbo myles]# gdisk /dev/sdd
GPT fdisk (gdisk) version 0.8.8

Partition table scan:
MBR: MBR only
BSD: not present
APM: not present
GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************


Warning! Secondary partition table overlaps the last partition by
5 blocks!
You will need to delete this partition or resize it in another utility.

Command (? for help):

So does this mean that it's not good - especially the bit about overlapping blocks? Does this mean I'll have to delete all my backups etc. from here and upgrade it to GPT and re-partition? Or can I just upgrade it to GPT without losing my data?

[artificeprime]

So does this mean that it's not good - especially the bit about overlapping blocks? Does this mean I'll have to delete all my backups etc. from here and upgrade it to GPT and re-partition? Or can I just upgrade it to GPT without losing my data?

IMO, the answers to your questions are: yes (it's not good); maybe (but if you have sufficient space on your 4TB to copy them, do so — there's no point in risking data loss — rsync is a wonderful CLI utility for this btw); and possibly, but the overlapping table may cause a problem (see below) unless you adjust the size of the offending partition (as gdisk instructed) first.

As for the bit about overlapping blocks: I honestly have no idea what that's about. Checking four of my own MBR-schemed disks I see one of them exhibits the same symptoms:

Code: Select all

Warning! Secondary partition table overlaps the last partition by 33 blocks!

Unfortunately, neither googling nor sifting through man pages give me a definitive answer as to whether or not this is really a bad thing. If I examine the drive via fdisk there's no warning or complaint about overlaps whatsoever. Also, I can't find a single tool in our Linux inventory to display a partition start and end in blocks (sectors and cylinders, yes, blocks, no).

Though I can't confirm, I believe the "secondary" partition table refers to the MBR scheme's extended partition. If the drive has more than one partition (and therefore at least one "logical" partition within the "extended") the secondary (extended) partition table might overlap with the edge of the extended partition itself. Non-detrimental to the MBR scheme, but problematic for GPT since it has no use for extended partitions and their "secondary" tables.

Bottom line, and this is just a guess (so take it for what little it's worth), is that in order to convert your drive from MBR to GPT the secondary (extended) partition table will have to be nuked, rendering any logical partitions (sdd5, sdd6, sdd7, etc.) you have within it inaccessible without disk recovery.

[artificeprime]

UPDATE: Disregard my guesswork above. I just couldn't leave this alone and tried a more direct Google query that got me the right answer pretty quick. We know we can trust it because it comes approved by our friend Rod Smith.

So all you have to do is shrink the drive's very last partition by a mere 5 blocks. Stupidly that's either as little as 2.5kB or at most 20kB depending on the size of its physical sectors. It would suck to nuke a whole partition just because of that.

What do you get if you run the following command as root?

Code: Select all

blockdev --getbsz /dev/sdd

[Myles]

artificeprime, mate, thanks for all of the effort you are putting in trying to help me, I really appreciate it!
Okay, here's the thing with the MyBook 3Tb. Under Windows Computer Management it shows (well you can just make out the first part of the partition size behind the open dialogue box!) the almost 3Tb partition but the option to convert to a GPT drive is greyed out!

MyBook Under Computer Disk Mgt.png (35.42 KiB) Viewed 3583 times

Also under Partition Manager (from Paragon Software), it shows the same thing - an almost 3Tb partition.

MyBook from Partition Mgr.png (28.61 KiB) Viewed 3583 times

So it has a valid 3Tb NTFS partition on it, but it seems to be MBR not GPT. This is going to be tricky and I'm going to have to pore over Rod's excellent pages on how to rectify this. It seems to me I 'm going to have to stop all backups going to it, move all of the relevant data off it before dealing with it!
The reason I bought the 4Tb was so my data wasn't locked into the MyBook, because if the MyBook interface fails, you can't plug the inside drive into another enclosure as WD has some sort of encryption or something so I didn't want that to happen. So I bought a stock WD 4Tb drive and an external enclosure to house it.

[artificeprime]

artificeprime, mate, thanks for all of the effort you are putting in trying to help me, I really appreciate it!

No problem. I often feel guilty for taking more from the FOSS community than I give back, so helping out on the occasional forum thread is the least I can do. That, and after our most recent exchange I wanted to figure out how to clear that overlap warning off my own drive (which I did, and it was pretty simple too), so I guess my motives aren't entirely altruistic.

As for your drive, at this point I'm not sure you should do anything about it after all. I think we were still misunderstanding each other, but your Window's screenshots cleared that up. I thought you couldn't get the My Book reading past the 2TiB MBR boundary (which would be an actual problem), but now I suspect you've simply fallen victim to an old, confusing, computer-product-marketing trap. You're not missing any space off either drive. Really.

According to your Window's screenshots your NexStar has ~3.7TB of its 4TB usable and your My Book has ~2.7TB of its 3TB usable. Both of those are correct. If you go back up thread and review the screen shots of my Go Flex (sdc) it too has ~2.7TB of its 3TB usable. This is normal. You never get the capacity it says on the packaging (if you still have the box for the NexStar, look on the bottom or one of the sides, there'll be a disclaimer in fine print about this so you can't sue for product misrepresentation). To you, a base-10 counting human, a "Tera-" equals 1,000,000,000,000 (or 10^12), but to a computer, a base-2 counting machine, a "Tera-" (now properly a "Tebi-") equals 1,099,511,627,776 (or 2^40). You bought a device in a box that advertised it would provide 3TB of storage and that's (almost) exactly what Western Digital sold you:

3,000,000,000,000 bytes = 3 TB (to you) = 2.728... TiB (to the machine).

Now partitioning the thing takes another 2,048 sectors (1 to 8 MiB) off the usable capacity for the partition table, and then formatting to a journalised file system (ext4 or NTFS) may take off another 1% for the journal, file table (i-node for ext4 or MFT for NTFS), and any other meta-data scratch space the OS reserves on the partition.

So if you do the math on a hypothetical 3TB drive you get:

Code: Select all

  3,000,000,000,000 bytes (3TB just as advertised)
–         8,388,608 bytes (8 MiB partition table assuming 4096 byte physical sectors)
= 2,999,991,611,392 bytes
×                99% (for what's left over after formatting to a journalised FS)
= 2,969,991,695,278 bytes (rounded off) actual usable capacity
=     2,900,382,515 KiB
=         2,832,405 MiB
=             2,766 GiB
=                 2.7 TiB (2.70119 rounded off to 1 decimal place)

If you compare this hypothetical drive (2,766 GiB) to yours (2794.4 GiB) it comes pretty close. Either your drive is slightly larger or (more likely) NTFS reserves a scratch space smaller than 1% which saves you nearly 28GiB off my estimate.

Unfortunately its often difficult to know which numeric base is implied. Does KB mean 1,000 or 1,024? Microsoft, MCC, some CLI tools, and even me (throughout most of this thread) can be pretty ambiguous. KDE on the other hand is always clear. The rule of thumb I've gone with for years now is that a computer-related number on product packaging is base-10 while anything the computer reports internally is base-2.

Anyway, your drive looks good. You've already filled it above the 2TiB boundary even with it partitioned on the MBR scheme. This confirms my suspicion that its an Advanced Format drive with 4096 byte physical sectors which it may be mapping to 512 byte logical ones. At this point I'd say converting it to GPT is purely optional and I'm not sure you'd get any benefit out of the switch.

[Myles]

I should have let you know earlier that I used to be in IT (20+ years) before I retired a year and a half ago, so I am fully conversant with the sleight-of-hand of the drive manufacturers re: the sizing of the drives no longer being 1,024 bytes/Gb but 1,0000/Gb!

Yes, it seems as though everything is working on the 3Tb drive as far as the ~2.7Tb partition so maybe I shouldn't muck around with it and just leave it as it is!

Again, thanks for all the help.

[artificeprime]

I should have let you know earlier that I used to be in IT (20+ years) before I retired a year and a half ago, so I am fully conversant with the sleight-of-hand of the drive manufacturers....

LOL! Okay, sorry about that.

That's a problem with the anonymous nature of web forums, there's no direct way of knowing if the other members are 17 or 70, or what their level of experience/competence is. Usually I proceed on the assumption that everyone else around here is an old UNIX wizard whom I can't hold a match stick to, but when you gave me the impression that you thought your drive wasn't being fully utilized, I jumped to the the wrong conclusion. I didn't mean to patronize you. Apologies again.

Just for the record, I'm in my mid-thirties, and though I'm not currently working in the field, I do have a college diploma as an IT tech. Supposedly I know a thing or two, but most days lately I wonder if I know more than nothing about anything. Thank goodness for Google!

Oh, and congrats on your retirement.

[Myles]

I am only in my mid 50's, I worked for 30+ years for the Federal Government. I retired early. I am in a superannuation fund that has a quirk - if you resign just before you turn 55 and take your money and pension after you turn 55....you actually get more in pension money!!!! That's why they closed the scheme to new people back in the early 90's! So I was set to do that, when the government wanted to save money and cut the budgets to all Federal departments. To cope with this the departments needed to lose staff, so my department offered voluntary redundancy, and before the secretary had finished her speech to us all, my email was winging it's way to personnel via my Blackberry! So eventually I got the VR (and left in June 2012) - over 12 months pay (because I had been in over 20 years) + my 3 years of annual leave (60 days) + my 9.5 months long-service leave. So a major payout! I had been purposely keeping my annual leave up high for just such an occasion, we were only supposed to have a maximum of 2 years! So we had to survive without my pay for 6 months (just my still wife working 4 days a week) till I could claim my money and pension.
So here I am, retired early.