Are we able to add persistence to live USBs this year?

This forum is dedicated to new ideas, suggestions and proposals.

Are we able to add persistence to live USBs this year?

Postby ca-on-adam » Apr 3rd, '18, 04:09

Hi,

I recently got interested in the capability to have persistent storage available in a live USB flash drive. This has been tossed around for a few years, and I would like to commit to doing what I can to find out how this can be implemented. I could prepare a road map of what help is needed for specific parts of this project, even if I don't have the skills to finish the entire thing myself.

Here are previous discussions on the subject:

2011
  1. https://forums.mageia.org/en/viewtopic.php?f=4&t=757
  2. https://forums.mageia.org/en/viewtopic.php?f=7&t=1316


2013
  1. https://forums.mageia.org/en/viewtopic.php?f=7&t=4980
  2. https://forums.mageia.org/en/viewtopic.php?f=7&t=5090

As I make progress poking around in Live ISOs from other distributions which support this, I will report back with a better understanding of what exactly needs to be done to get this working. I'd like to help get it done this year, ideally in time for Mageia 7.
ca-on-adam
 
Posts: 49
Joined: Aug 31st, '11, 13:33

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 4th, '18, 09:43

I've been meaning to look into this for a while. A quick search with Google suggest there are two types of persistence:
  • a separate partition on the USB stick that you can use to save files
  • persistent backing memory for the root filesystem
The latter seems the more useful to me. A quick experiment last night suggests it's fairly easy to implement. The only obstacle is that to make it work, I needed to remove an old bug fix (bug 3334). I suspect that bug fix is no longer needed, but as it only affected some users, it's hard to be sure without widespread testing. But that's what alpha releases are for...

If it wasn't for that bug fix, you'd be able to implement the first type of persistence with the current Live ISOs.
martinw
 
Posts: 392
Joined: May 14th, '11, 10:59

Re: Are we able to add persistence to live USBs this year?

Postby morgano » Apr 4th, '18, 21:26

A third, easy, way is to boot the live system and actually install it to a USB key, keeping all drivers at the question dialog, so it is prepared to meet different systems.

I guess that when putting it in a new system, you may need to change the video driver, maybe more, but when put in same system i guess it boots quicker than a live .iso.
Mandriva since 2006, then Mageia since 2011 at home & work. Thinkpad T40 T42p T43 T60 T61 T400. Aspire 7. Fileserver. Workstation using LVM, LUKS, VirtualBox, BOINC, Dropbox, CAD, urpmi-proxy, NextCloud...
morgano
 
Posts: 487
Joined: Jun 15th, '11, 17:51
Location: Kivik, Sweden

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 4th, '18, 22:56

morgano wrote:I guess that when putting it in a new system, you may need to change the video driver, maybe more, but when put in same system i guess it boots quicker than a live .iso.

It should boot quicker, yes. The downside is that it takes more space, because the root filesystem is uncompressed.
martinw
 
Posts: 392
Joined: May 14th, '11, 10:59

Re: Are we able to add persistence to live USBs this year?

Postby morgano » Apr 5th, '18, 00:53

Yes and uncompressed is probably slower too, as i think a normal CPU decompress faster than a USB read - and likewise for write. But maybe not for random access. So two win points for compressed systems.
I wonder if it could install to compressed btrfs filesystem? Or other solution. This will save space over a compressed read-only system + uncompressed overlay when large updates, or additions are made.
Mandriva since 2006, then Mageia since 2011 at home & work. Thinkpad T40 T42p T43 T60 T61 T400. Aspire 7. Fileserver. Workstation using LVM, LUKS, VirtualBox, BOINC, Dropbox, CAD, urpmi-proxy, NextCloud...
morgano
 
Posts: 487
Joined: Jun 15th, '11, 17:51
Location: Kivik, Sweden

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 5th, '18, 09:47

morgano wrote:I wonder if it could install to compressed btrfs filesystem? Or other solution. This will save space over a compressed read-only system + uncompressed overlay when large updates, or additions are made.

I've not tried this, but if you use Custom disk partitioning, select Expert mode, set the partition type to btrfs, and add 'compress' to the mount options (under Options->Advanced), you should get a compressed btrfs filesystem. See the btrfs man page for more details.

I have also thought about writing a small utility that would let you run the draklive installer from a loopback-mounted ISO file, to avoid the intermediate copy to and boot from a USB stick. But there are some more important things to be done first.
martinw
 
Posts: 392
Joined: May 14th, '11, 10:59

Re: Are we able to add persistence to live USBs this year?

Postby morgano » Apr 5th, '18, 21:45

Writing some ideas down while thinking. /!\ Warning: i do not know much about this - these ideas are not researched much...

Interesting! Will try compressed btrfs on a stick some time :)
I guess it need a separate /boot partition in i.e ext4 format, if system root is in compressed btrfs?
BTW myself would also make a ntfs partition so the same stick can also be used to simply plug into a MS system and exchange files.
And in my case, as i would probably carry some of my customers files i need to encrypt , so i would make a LUKS encrypted LVM, and the compressed btrfs in that. Possible also swap partition in that LVM, but see further below.
---
To be able to boot many different systems, the install should have many drivers... is there an easy way to include all GPU and wifi etc drivers. Something like task-graphicsdrivers, task-wifidrivers, etc would be handy.
Do the live iso run some hardware detection and configuring that an installed system skips over?
- If so, can it be set on this USB-installed system to run every boot, or maybe by grub menu choice?
---
Regarding booting on most systems, can Mageia be installed so it boots like the Live systems; both on UEFI systems that require UEFI, and on machines that only do old style BIOS boot?
-And when updating kernel, both boot methods are updated?
Sounds hard.
---
Another idea: enable zswap to eliminate/minimise swap usage to USB, but still use USB swap automatically when - if ever - RAM is not enough. (it is a compressed cache layer on top of the swap device - especially note: the total available swap is not larger than the swap on disk.) https://en.wikipedia.org/wiki/Zswap https://wiki.archlinux.org/index.php/Zswap It can easily be a separate boot choice in grub, adding kernel parameter zswap.enabled=1, or can be enabled in runtime. Without zswap enabled the system simply use the disk swap space.
-
BTW, it would be good if the swap on USB was compressed.
So maybe create the USB swap as a file on the already compressed file system?
BUT! [url}https://btrfs.wiki.kernel.org/index.php/Project_ideas#Swap_file_support] btrfs do not support swap files[/URL]. -> Need some more complicated setup. Loopback, with some compression mojo on a file on my ntfs partition. seems too much work and too much that can break...

So i would maybe put them in my ntfs partition. Uh, if that is compatible?

https://support.rackspace.com/how-to/cr ... swap-file/ https://www.cyberciti.biz/faq/linux-add ... ile-howto/ That also makes it easier to modify the size, than if we use separate partition. If we create several small swap files to begin with, we can delete one at a time if we need more file space for other stuff and the result is immediate without partitioning hassles (but dont forget swapoff and edit fstab...). Priority could be used to steer kernel to first use /swapfile0, then /swapfile1, etc So i would put them in my ntfs partition. Uh, if that is compatible?
-
One stupid thing with this setup is that when zswap is about to move compressed content in RAM to disk, it decompress it first... Normally that is good i think because it most often make read back quick = less delay. In this setup data will be compressed and decompressed... Oh well...
---
An alternative is zram, but it use RAM only - a problem when we do not know how much memory the target system have. USBsticks are rather cheap so having extra GB of SWAP i think is normally not a problem. Also, zram can not be so easily configured as a kernel boot choice.
---
About what stick to use:
I have noted flash sticks can identify them selves in different ways, and some computers refuse to boot on some sticks. (same iso works on another stick)
Fast, good quality, that support both USB3 and USB2. Large size for storing much, and large also normally mean it is faster to write to, and have better write endurance measured in bytes written.
Mechanically sturdy. Maybe connect using a short extension - less risk of bending connector (been there!) in stick or laptop. (of good quality - i have had glitchy ones...)
---
32-bit system - so it can boot on both 32 and 64 bit.
OR, is it possible to have both 32 and 64 bit kernel selectable in grub, or automatically?
Mandriva since 2006, then Mageia since 2011 at home & work. Thinkpad T40 T42p T43 T60 T61 T400. Aspire 7. Fileserver. Workstation using LVM, LUKS, VirtualBox, BOINC, Dropbox, CAD, urpmi-proxy, NextCloud...
morgano
 
Posts: 487
Joined: Jun 15th, '11, 17:51
Location: Kivik, Sweden

Re: Are we able to add persistence to live USBs this year?

Postby martinw » Apr 7th, '18, 00:53

morgano wrote:Interesting! Will try compressed btrfs on a stick some time :)
I guess it need a separate /boot partition in i.e ext4 format, if system root is in compressed btrfs?

I would guess so to.
To be able to boot many different systems, the install should have many drivers... is there an easy way to include all GPU and wifi etc drivers. Something like task-graphicsdrivers, task-wifidrivers, etc would be handy.
Do the live iso run some hardware detection and configuring that an installed system skips over?
- If so, can it be set on this USB-installed system to run every boot, or maybe by grub menu choice?

If you don't select "Remove unneeded hardware support", all non-proprietary GPU and WiFi drivers will be there in the installed system. If you install the proprietary driver packages before running the live installer, you'll get those too. The only exception is the two Broadcom proprietary wireless support packages, which conflict, so you can't install them both at the same time.

service_harddrake is run during boot, and is responsible for hardware detection and (where necessary) driver installation. I don't know how well it copes with a complete change of hardware - you might need to delete its record of what it found before. I know that is done when creating the Live ISOs.
Regarding booting on most systems, can Mageia be installed so it boots like the Live systems; both on UEFI systems that require UEFI, and on machines that only do old style BIOS boot?
-And when updating kernel, both boot methods are updated?
Sounds hard.

It's not hard to set up the dual MBR/UEFI boot manually, but drakboot doesn't support it, and hence only the current boot menu will get updated when you update a kernel.
OR, is it possible to have both 32 and 64 bit kernel selectable in grub, or automatically?

I tried that recently on a 32-bit system, but urpmi wouldn't let me install the 64-bit kernel.
martinw
 
Posts: 392
Joined: May 14th, '11, 10:59


Return to Ideas and suggestions

Who is online

Users browsing this forum: No registered users and 1 guest

cron