Are we able to add persistence to live USBs this year?

This forum is dedicated to new ideas, suggestions and proposals.

Are we able to add persistence to live USBs this year?

Postby ca-on-adam » Apr 3rd, '18, 04:09

Hi,

I recently got interested in the capability to have persistent storage available in a live USB flash drive. This has been tossed around for a few years, and I would like to commit to doing what I can to find out how this can be implemented. I could prepare a road map of what help is needed for specific parts of this project, even if I don't have the skills to finish the entire thing myself.

Here are previous discussions on the subject:

2011
  1. https://forums.mageia.org/en/viewtopic.php?f=4&t=757
  2. https://forums.mageia.org/en/viewtopic.php?f=7&t=1316


2013
  1. https://forums.mageia.org/en/viewtopic.php?f=7&t=4980
  2. https://forums.mageia.org/en/viewtopic.php?f=7&t=5090

As I make progress poking around in Live ISOs from other distributions which support this, I will report back with a better understanding of what exactly needs to be done to get this working. I'd like to help get it done this year, ideally in time for Mageia 7.
ca-on-adam
 
Posts: 57
Joined: Aug 31st, '11, 13:33

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 4th, '18, 09:43

I've been meaning to look into this for a while. A quick search with Google suggest there are two types of persistence:
  • a separate partition on the USB stick that you can use to save files
  • persistent backing memory for the root filesystem
The latter seems the more useful to me. A quick experiment last night suggests it's fairly easy to implement. The only obstacle is that to make it work, I needed to remove an old bug fix (bug 3334). I suspect that bug fix is no longer needed, but as it only affected some users, it's hard to be sure without widespread testing. But that's what alpha releases are for...

If it wasn't for that bug fix, you'd be able to implement the first type of persistence with the current Live ISOs.
martinw
 
Posts: 609
Joined: May 14th, '11, 10:59

Re: Are we able to add persistence to live USBs this year?

Postby morgano » Apr 4th, '18, 21:26

A third, easy, way is to boot the live system and actually install it to a USB key, keeping all drivers at the question dialog, so it is prepared to meet different systems.

I guess that when putting it in a new system, you may need to change the video driver, maybe more, but when put in same system i guess it boots quicker than a live .iso.
Mandriva since 2006, Mageia 2011 at home & work. Thinkpad T40, T43, T400, T510, Dell M4400, M6300, Acer Aspire 7. Workstation using LVM, LUKS, VirtualBox, BOINC
morgano
 
Posts: 1306
Joined: Jun 15th, '11, 17:51
Location: Kivik, Sweden

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 4th, '18, 22:56

morgano wrote:I guess that when putting it in a new system, you may need to change the video driver, maybe more, but when put in same system i guess it boots quicker than a live .iso.

It should boot quicker, yes. The downside is that it takes more space, because the root filesystem is uncompressed.
martinw
 
Posts: 609
Joined: May 14th, '11, 10:59

Re: Are we able to add persistence to live USBs this year?

Postby morgano » Apr 5th, '18, 00:53

Yes and uncompressed is probably slower too, as i think a normal CPU decompress faster than a USB read - and likewise for write. But maybe not for random access. So two win points for compressed systems.
I wonder if it could install to compressed btrfs filesystem? Or other solution. This will save space over a compressed read-only system + uncompressed overlay when large updates, or additions are made.
Mandriva since 2006, Mageia 2011 at home & work. Thinkpad T40, T43, T400, T510, Dell M4400, M6300, Acer Aspire 7. Workstation using LVM, LUKS, VirtualBox, BOINC
morgano
 
Posts: 1306
Joined: Jun 15th, '11, 17:51
Location: Kivik, Sweden

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 5th, '18, 09:47

morgano wrote:I wonder if it could install to compressed btrfs filesystem? Or other solution. This will save space over a compressed read-only system + uncompressed overlay when large updates, or additions are made.

I've not tried this, but if you use Custom disk partitioning, select Expert mode, set the partition type to btrfs, and add 'compress' to the mount options (under Options->Advanced), you should get a compressed btrfs filesystem. See the btrfs man page for more details.

I have also thought about writing a small utility that would let you run the draklive installer from a loopback-mounted ISO file, to avoid the intermediate copy to and boot from a USB stick. But there are some more important things to be done first.
martinw
 
Posts: 609
Joined: May 14th, '11, 10:59

Re: Are we able to add persistence to live USBs this year?

Postby morgano » Apr 5th, '18, 21:45

Writing some ideas down while thinking. /!\ Warning: i do not know much about this - these ideas are not researched much...

Interesting! Will try compressed btrfs on a stick some time :)
I guess it need a separate /boot partition in i.e ext4 format, if system root is in compressed btrfs?
BTW myself would also make a ntfs partition so the same stick can also be used to simply plug into a MS system and exchange files.
And in my case, as i would probably carry some of my customers files i need to encrypt , so i would make a LUKS encrypted LVM, and the compressed btrfs in that. Possible also swap partition in that LVM, but see further below.
---
To be able to boot many different systems, the install should have many drivers... is there an easy way to include all GPU and wifi etc drivers. Something like task-graphicsdrivers, task-wifidrivers, etc would be handy.
Do the live iso run some hardware detection and configuring that an installed system skips over?
- If so, can it be set on this USB-installed system to run every boot, or maybe by grub menu choice?
---
Regarding booting on most systems, can Mageia be installed so it boots like the Live systems; both on UEFI systems that require UEFI, and on machines that only do old style BIOS boot?
-And when updating kernel, both boot methods are updated?
Sounds hard.
---
Another idea: enable zswap to eliminate/minimise swap usage to USB, but still use USB swap automatically when - if ever - RAM is not enough. (it is a compressed cache layer on top of the swap device - especially note: the total available swap is not larger than the swap on disk.) https://en.wikipedia.org/wiki/Zswap https://wiki.archlinux.org/index.php/Zswap It can easily be a separate boot choice in grub, adding kernel parameter zswap.enabled=1, or can be enabled in runtime. Without zswap enabled the system simply use the disk swap space.
-
BTW, it would be good if the swap on USB was compressed.
So maybe create the USB swap as a file on the already compressed file system?
BUT! [url}https://btrfs.wiki.kernel.org/index.php/Project_ideas#Swap_file_support] btrfs do not support swap files[/URL]. -> Need some more complicated setup. Loopback, with some compression mojo on a file on my ntfs partition. seems too much work and too much that can break...

So i would maybe put them in my ntfs partition. Uh, if that is compatible?

https://support.rackspace.com/how-to/cr ... swap-file/ https://www.cyberciti.biz/faq/linux-add ... ile-howto/ That also makes it easier to modify the size, than if we use separate partition. If we create several small swap files to begin with, we can delete one at a time if we need more file space for other stuff and the result is immediate without partitioning hassles (but dont forget swapoff and edit fstab...). Priority could be used to steer kernel to first use /swapfile0, then /swapfile1, etc So i would put them in my ntfs partition. Uh, if that is compatible?
-
One stupid thing with this setup is that when zswap is about to move compressed content in RAM to disk, it decompress it first... Normally that is good i think because it most often make read back quick = less delay. In this setup data will be compressed and decompressed... Oh well...
---
An alternative is zram, but it use RAM only - a problem when we do not know how much memory the target system have. USBsticks are rather cheap so having extra GB of SWAP i think is normally not a problem. Also, zram can not be so easily configured as a kernel boot choice.
---
About what stick to use:
I have noted flash sticks can identify them selves in different ways, and some computers refuse to boot on some sticks. (same iso works on another stick)
Fast, good quality, that support both USB3 and USB2. Large size for storing much, and large also normally mean it is faster to write to, and have better write endurance measured in bytes written.
Mechanically sturdy. Maybe connect using a short extension - less risk of bending connector (been there!) in stick or laptop. (of good quality - i have had glitchy ones...)
---
32-bit system - so it can boot on both 32 and 64 bit.
OR, is it possible to have both 32 and 64 bit kernel selectable in grub, or automatically?
Mandriva since 2006, Mageia 2011 at home & work. Thinkpad T40, T43, T400, T510, Dell M4400, M6300, Acer Aspire 7. Workstation using LVM, LUKS, VirtualBox, BOINC
morgano
 
Posts: 1306
Joined: Jun 15th, '11, 17:51
Location: Kivik, Sweden

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 7th, '18, 00:53

morgano wrote:Interesting! Will try compressed btrfs on a stick some time :)
I guess it need a separate /boot partition in i.e ext4 format, if system root is in compressed btrfs?

I would guess so to.
To be able to boot many different systems, the install should have many drivers... is there an easy way to include all GPU and wifi etc drivers. Something like task-graphicsdrivers, task-wifidrivers, etc would be handy.
Do the live iso run some hardware detection and configuring that an installed system skips over?
- If so, can it be set on this USB-installed system to run every boot, or maybe by grub menu choice?

If you don't select "Remove unneeded hardware support", all non-proprietary GPU and WiFi drivers will be there in the installed system. If you install the proprietary driver packages before running the live installer, you'll get those too. The only exception is the two Broadcom proprietary wireless support packages, which conflict, so you can't install them both at the same time.

service_harddrake is run during boot, and is responsible for hardware detection and (where necessary) driver installation. I don't know how well it copes with a complete change of hardware - you might need to delete its record of what it found before. I know that is done when creating the Live ISOs.
Regarding booting on most systems, can Mageia be installed so it boots like the Live systems; both on UEFI systems that require UEFI, and on machines that only do old style BIOS boot?
-And when updating kernel, both boot methods are updated?
Sounds hard.

It's not hard to set up the dual MBR/UEFI boot manually, but drakboot doesn't support it, and hence only the current boot menu will get updated when you update a kernel.
OR, is it possible to have both 32 and 64 bit kernel selectable in grub, or automatically?

I tried that recently on a 32-bit system, but urpmi wouldn't let me install the 64-bit kernel.
martinw
 
Posts: 609
Joined: May 14th, '11, 10:59

Re: Are we able to add persistence to live USBs this year?

Postby ca-on-adam » Apr 21st, '18, 20:09

martinw wrote:...
  • persistent backing memory for the root filesystem
The latter seems the more useful to me...

I was thinking of that type. The Live ISO image would be raw written to the USB drive first, and the overlay data would be in a partition (or raw area?) following that. It would let you have the feel of being able to save files in the live user's home folder as well as run updates to the software. Also by deleting the overlay data you can revert to the exact original Live system without re-imaging the USB drive.

From digging around in an initramfs from a live ISO known to support persistence (can't remember if it was Debian-Live or Fedora), it looks like the filesystem type is "overlayfs"



morgano wrote:... /!\ Warning: i do not know much about this...

That's fine, neither do I. I am willing to learn though, as this is a labour of love.

morgano wrote:...
Do the live iso run some hardware detection and configuring that an installed system skips over?
- If so, can it be set on this USB-installed system to run every boot, or maybe by grub menu choice?

The Live ISO images will do hardware detection, yes. And if the ISO is raw-written to the USB drive, it will work fine.

morgano wrote:...both on UEFI systems that require UEFI, and on machines that only do old style BIOS boot?

The ISO images for the Live systems are set to be able to auto-detect whether to boot UEFI or BIOS(MBR style) As long as the image is raw-written, this will work.

morgano wrote:-And when updating kernel, both boot methods are updated?

That is a good question, The overlay won't be accessible until after the kernel is loaded, so that means that updating the kernel, or grub, or anything that needs to run at boot time will have no effect. The ISO bootloader will still see the original / old version kernel and initrd. After booting, the overlayfs is included and you will see the new kernel and initrd in the filesystem, but it won't be the running kernel.

A possible problem is that if the overlayfs records a deletion of the original kernel and associated files (I don't know what they all are) that could cause problems where any of the associated files are needed while the system is running, but are made to seem like they are deleted by the effect of the changed overlayfs. I will try to see if other distributions have done anything to avoid this.

morgano wrote:...
So i would maybe put them in my ntfs partition. Uh, if that is compatible?

I don't know what the overlayfs looks like. If it is a raw area that takes up the entire rest of the USB drive, then no. But if the overlayfs can live in partition #2, then you could add an NTFS partition as #3.

morgano wrote:...some computers refuse to boot on some sticks...

That is a hardware issue, nothing can be done with the bootloader on the USB drive to fix that.

morgano wrote:32-bit system - so it can boot on both 32 and 64 bit.
OR, is it possible to have both 32 and 64 bit kernel selectable in grub, or automatically?

I would use the same strategy as advised for the LiveCDs: if you are only running on 64-bit hardware, use the 64-bit ISO. If you might be using both 32-bit and 64-bit hardware, then you would use the 32-bit ISO. Inclusion of extra kernels would be the responsibility of the Live system maintainers, this project I am proposing is not about changing the LiveCD contents.[/quote]


Thanks for the response everyone, I am feeling more optimistic about this!


* In order to find out about distributions Live ISOs which persistence works in, I referred to the online vendor OSDisc.com, for a quick reference in their USB drive listings.
ca-on-adam
 
Posts: 57
Joined: Aug 31st, '11, 13:33

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 21st, '18, 23:02

ca-on-adam wrote:
martinw wrote:...
  • persistent backing memory for the root filesystem
The latter seems the more useful to me...

I was thinking of that type. The Live ISO image would be raw written to the USB drive first, and the overlay data would be in a partition (or raw area?) following that.

It needs to be a formatted partition. Annoyingly, both drakdisk and gparted can't be used to add the partition - they see the ISO9660 filesystem and assume it's an optical disk. So at the moment you have to use fdisk and mkfs.
ca-on-adam wrote:From digging around in an initramfs from a live ISO known to support persistence (can't remember if it was Debian-Live or Fedora), it looks like the filesystem type is "overlayfs"

Yes, all recent Mageia Live ISOs use overlayfs.
ca-on-adam wrote:
morgano wrote:...
Do the live iso run some hardware detection and configuring that an installed system skips over?
- If so, can it be set on this USB-installed system to run every boot, or maybe by grub menu choice?

The Live ISO images will do hardware detection, yes. And if the ISO is raw-written to the USB drive, it will work fine.

When I tested this, first booting my experimental persistant ISO on my laptop, then on my desktop, hardware detection worked fine. The only quirk I noticed was that network initialisation scripts are created in /etc/sysconfig/network-scripts based on the network device name. When booting on the desktop, the appropriately-named script was created, but the old scripts were left behind. This caused the system to try (and fail) to start those network interfaces. I can see why harddrake doesn't delete the old scripts - it has no way of knowing they aren't something the user has created.
ca-on-adam wrote:A possible problem is that if the overlayfs records a deletion of the original kernel and associated files (I don't know what they all are) that could cause problems where any of the associated files are needed while the system is running, but are made to seem like they are deleted by the effect of the changed overlayfs. I will try to see if other distributions have done anything to avoid this.

The most important files would be the kernel modules in /lib/modules. If these vanished when the overlay was mounted, the system would almost certainly fail to boot. I wonder if there's a way to instruct overlayfs that certain files/directories must be preserved...
ca-on-adam wrote:I don't know what the overlayfs looks like. If it is a raw area that takes up the entire rest of the USB drive, then no. But if the overlayfs can live in partition #2, then you could add an NTFS partition as #3.

The way I've implemented it, the script that mounts the overlay searches for a partition labelled "Mageia-Persist" on the same device as the base filesystem. So you can allocate the remaining space on the USB stick any way you like. There's probably a lower limit on the size for the persistance partition, below which things won't work too well.
martinw
 
Posts: 609
Joined: May 14th, '11, 10:59

Re: Are we able to add persistence to live USBs this year?

Postby papoteur » Jan 2nd, '19, 16:34

Hello,
Thanks to Martin, this is now working since 6.1 Live images. The additional partition on the USB stick has to be named "mgalive-persist".
Isodumper now in cauldron manages this new feature when a box is ticked, and creates a partition in the remaining space on the stick after the write operation.
papoteur
 
Posts: 93
Joined: Oct 27th, '11, 22:28

Re: Are we able to add persistence to live USBs this year?

Postby doktor5000 » Jan 2nd, '19, 17:35

That sounds awesome :D

Would be greatly appreciated if you could add a short note about that to https://wiki.mageia.org/en/Mageia_7_Release_Notes somewhere, at least for isodumper and maybe in the Available installation media section.
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17629
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: Are we able to add persistence to live USBs this year?

Postby morgano » Jan 4th, '19, 00:31

Yes - and Isodumper on Mageia 6 should also be able to updated to achieve persistence when making USB sticks of Mageia 6.1 and 7+ isos :)
https://bugs.mageia.org/show_bug.cgi?id=23654#c14

EDIT: New page: https://wiki.mageia.org/en/Persistent_live_systems
Mandriva since 2006, Mageia 2011 at home & work. Thinkpad T40, T43, T400, T510, Dell M4400, M6300, Acer Aspire 7. Workstation using LVM, LUKS, VirtualBox, BOINC
morgano
 
Posts: 1306
Joined: Jun 15th, '11, 17:51
Location: Kivik, Sweden


Return to Ideas and suggestions

Who is online

Users browsing this forum: No registered users and 1 guest

cron