[DONE] Need faster rpmdrake, urpmi, rpm, mirrors

This forum is dedicated to new ideas, suggestions and proposals.

[DONE] Need faster rpmdrake, urpmi, rpm, mirrors

Postby agron » Sep 26th, '15, 07:23

Since I am doing a lot installs and upgrades I find the current standard package management tools to be very slow and I am looking for ways to speed up this process.

I have itemized parts of the process that seem slow to me as follows:

A. rpmdrake
1. Dialog: Please wait, finding available packages...
2. Status bar: Please wait, listing packages...
3. Getting files from xml meta-data
3. Getting changelog from xml meta-data

B. Urpmi
1. Preparing....

C. Mirrors
1. Downloading


A. rpmdrake
Rpmdrake seems to be slow in just getting the names of packages for the first part. It is probably spawning other processes.
Is there a way to speedup this part? Are there alternatives?

As for the second part I think this just how Perl + GUI works. It could be that Perl is slow with arrays and the GUI is slow with maintaining the tree of categories.

B. Urpmi
I also noticed there is parallel version of urpmi in the repository, urpmi-parallel-ka-run - Parallel extensions to urpmi using ka-run​.
I have never used it. I would like to know if anyone had any experience with it. Ka-run seems to be part of machine clustering tools which I am not really in favor of right now.
On the other hand, Urpmi probably uses rpm which uses decompression tools for installation. I wonder if it uses the plain old single threaded gzip. Could we replace that with pigz?


C. Mirrors
To speed things in terms of downloading from around the world, I have created a local mirror inside my corporate environment. I will publish the steps here once I have time: https://wiki.mageia.org/en/Finding_Mage ... te_network 


originally posted as https://plus.google.com/+AgronSelimaj/posts/TRy3FSv4LdA
Last edited by agron on Sep 30th, '15, 00:55, edited 2 times in total.
agron
 
Posts: 15
Joined: Sep 4th, '14, 20:55

Re: Need faster rpmdrake, urpmi, rpm, mirrors

Postby doktor5000 » Sep 26th, '15, 23:30

agron wrote:A. rpmdrake
Rpmdrake seems to be slow in just getting the names of packages for the first part. It is probably spawning other processes.
Is there a way to speedup this part? Are there alternatives?

Yes we could drop a lot of packages from the repositories, so less packages need to be listed.
If it's too slow for you, then simply use urpmi and urpmq/urpmf which are a lot of faster.
In the past there was also smart as alternative to rpmdrake but it was unmaintained for several years.

agron wrote:As for the second part I think this just how Perl + GUI works. It could be that Perl is slow with arrays and the GUI is slow with maintaining the tree of categories.

Iterating through more then 10.000 packages metadata, and putting all of them in a tree structure simply takes some time.
You're comparing this to what package manager actually?

And FWIW, those two steps
3. Getting files from xml meta-data
3. Getting changelog from xml meta-data

are only run if you want to display the files or changelog from some package, they are not run in general.

agron wrote:B. Urpmi
I also noticed there is parallel version of urpmi in the repository, urpmi-parallel-ka-run - Parallel extensions to urpmi using ka-run​.
I have never used it. I would like to know if anyone had any experience with it. Ka-run seems to be part of machine clustering tools which I am not really in favor of right now.
On the other hand, Urpmi probably uses rpm which uses decompression tools for installation. I wonder if it uses the plain old single threaded gzip. Could we replace that with pigz?

urpmi-parallel-ka-run is for installing packages in parallel if you manage a lot of machines, that's not for standalone machines.
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17629
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: Need faster rpmdrake, urpmi, rpm, mirrors

Postby agron » Sep 28th, '15, 22:50

I agree. urpmi/q/f are way faster.
Code: Select all
urpmq --list -g -f > /dev/null
prints a list of 30500 packages in less than 2 seconds. drakrpm starts in 77 seconds. Huge difference.

Therefore, it makes me wonder if someone started working (or is willing to) in a urpmi/q/f GUI frontend with Qt and C++? Not as a replacement for rpmdrake, but as an additional tool, with less features, simple but fast to start, fast in finding and selecting packages for installation or removal.
Last edited by doktor5000 on Sep 29th, '15, 01:30, edited 1 time in total.
Reason: removed fullquote
agron
 
Posts: 15
Joined: Sep 4th, '14, 20:55

Re: Need faster rpmdrake, urpmi, rpm, mirrors

Postby doktor5000 » Sep 29th, '15, 01:40

Please next time do not use fullquotes, but only use the Reply function. Greatly improves clarity and reading flow ;)

And for the frontend, don't hesitate and start creating it. But I don't see the point, as urpm* toolsuite can be used greatly for scripting and such,
but why add a GUI for that, seems pretty useless to me. I mean, what commands are used often?
  • urpmf 'filename or regex_for_part_of_filename' to search which package contains a (part of a) filename
  • urpmf -f --name -m part_of_packagename to search a package by name (others prefer fuzzy search via urpmq -y but I don't)
  • urpmq -i package to show package metadata
  • urpmi package to install something
Why add a frontend for that, functionality is there?

For completeness sake, there has been some work on migrating the drakxtools away from perl and away from GTK to a toolkit-agnostic way of displaying things, using libyui.
It's called manatools and also features rpmdragora as an alternative to rpmdrake. See https://wiki.mageia.org/en/Mageia_5_Rel ... ls_preview for some details.

For searching files or managing orphans, there's also qt4urpm: https://wiki.mageia.org/en/Removing_pac ... ge_Orphans
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17629
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: Need faster rpmdrake, urpmi, rpm, mirrors

Postby agron » Sep 29th, '15, 18:19

GUI is required when you want to use the mouse or the touchscreen. Or if you just prefer GUI.

OMG 'mana rpm' from Manatools is extremely fast. It is so responsive. I love it.

Thank you so much doktor. I don't need to make anything now.
agron
 
Posts: 15
Joined: Sep 4th, '14, 20:55

Re: Need faster rpmdrake, urpmi, rpm, mirrors

Postby doktor5000 » Sep 30th, '15, 00:44

Please mark the thread accordingly by editing the topic of the first post and prefix it by [DONE], thanks
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17629
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: [DONE] Need faster rpmdrake, urpmi, rpm, mirrors

Postby benmc » Oct 8th, '15, 04:53

Maybe an extension of this.

Irregularly, when attempting an update, I get an "exit aria 2" or some other number and "installation of packages xyz failed, would you like to retry y/N".

I understand that this may be due to a failure of some sort at my localest mirror.
Is it possible to force the updater to automagically look for another mirror when this happens [or maybe a user input prompt window] and then continue with the update.
At the completion of the update then revert back to my localest mirror

I understand that with *buntu derivatives, before an update starts, mirrors are checked for availability.
benmc
 
Posts: 1175
Joined: Sep 2nd, '11, 12:45
Location: Pirongia, New Zealand

Re: [DONE] Need faster rpmdrake, urpmi, rpm, mirrors

Postby doktor5000 » Oct 8th, '15, 11:09

benmc wrote:I understand that this may be due to a failure of some sort at my localest mirror.
Is it possible to force the updater to automagically look for another mirror when this happens [or maybe a user input prompt window] and then continue with the update.
At the completion of the update then revert back to my localest mirror


That would be a good point for a Mageia 6 Feature, as that is missing since years.
Best put up a feature like explained in https://wiki.mageia.org/en/Features_policy
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17629
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: [DONE] Need faster rpmdrake, urpmi, rpm, mirrors

Postby benmc » Oct 9th, '15, 08:54

benmc
 
Posts: 1175
Joined: Sep 2nd, '11, 12:45
Location: Pirongia, New Zealand


Return to Ideas and suggestions

Who is online

Users browsing this forum: No registered users and 1 guest

cron