Troubleshooting Proxy Authentication for Mageia tools

[luzemario]

Hi guys. I am deploying Mageia 2 machines at my work, but I am experiencing several issues about proxy authentication with Mageia tools. Is not possible to update, install programs or upgrade system. Is not even possible to get mirrorlists or get XML descriptions. Downloaders receive HTTP error 407 (Proxy authentication required) all the time. It is not transparent to end users, since graphic tools hide too much information.

The only proxy authentication scheme allowed at my work is NTLM, since it is a Windows-only terrain. Networking runs smoothly for web browsing, but nothing else can download since downloaders cannot authenticate with proxy. Proxies are Cent OS 5 machines running Squid.

After trial and error I figured out CURL is the only downloader wich can handle NTLM authentication with --proxy-ntlm option. But Mageia scripts are not ready to save and pass parameters to downloader (not even the command-line ones).

Can someone discuss about this? I will start to fill bugs in bugzilla, but I want to search for alternatives/workarounds to fill the bug better.

[oj]

I would assume the IT admins don't want that sort of traffic inside the firewall. I'd tell someone who administers the proxy what it is I want to do and ask if they could open things up a bit for me.

[luzemario]

Here this is not a issue. But IT adminis aked me why open source software can't authenticate with open source software (since proxies are Squid/CentOS 5 machines...). They argued me to dig more and discover a solution. I think they are correct.

Digging a bit more I discovered tweaking urpmi.cfg and proxy.cfg helps a lot, but Mageia 2 graphical tools are buggy, and do not use config file directives as expected.

Actually I can issue a

Code: Select all

urpmi mc

and urpmi asks for user/password (and passes --proxy-ntlm parameter to curl), installing 'mc' package sucessfully, but graphical drakrpm cause curl to get a HTTP 407 (and curl shows a error 22 to interface without details).

In my honest opinion it is not a good behavior, since it has NO USEFUL INFO to the user about the cause of error.

[doktor5000]

Does

Code: Select all

urpmi --proxy YOURPROXYHERE:PORT --proxy-user=ask anypackagetoinstall

work?
Also check man urpmi.cfg, you can specify another default downloader and pass it some options directly (for curl/rsync/wget)
Hopefully you also checked man proxy.cfg ?

[luzemario]

Hi, Doktor!

Yes, I checked manuals for both proxy.cfg and urpmi.cfg. The main trouble is passing the option "--proxy-ntlm" to curl. Other downloaders/rsyncers can't authenticate with NTLM, I didn't figured out why.

urpmi command you suggested don't work. It don't follow all --proxy parameters. Curl can't authenticate and receives a HTTP 407.
urpmi does not ask for proxy user, despite you use --proxy-user=ask or not;
urpmi does not respect --downloader=curl parameter if current mirror is a rsync:// address and does not select another compatible mirror. It tries to use rsync every time in this case.

urpmi does follow urpmi.cfg data, and pass --proxy-ntlm I added to curl parameters;
urpmi does follow proxy.cfg data, and asks for proxy user if instructed to do so in proxy.cfg;
urpmi works as expected if urpmi.cfg and proxy.cfg both are configured correctly. But drakrpm bypasses curl parameters from urpmi.cfg file and makes curl get a HTTP 407. Drakrpm asks for proxy user if this is specified in proxy.cfg, but appears to not obey curl parameters in urpmi.cfg.

Drakrpm NEVER pass parameters to curl. There is no way to enter parameters to use with downloader in drakrpm.

[doktor5000]

Well, if you're repos are pointing to rsync mirrors, how do you expect curl to work? Either rsync or aria2 can do that, IINM.
But it's probably best to open a bug report for that. https://wiki.mageia.org/en/How_to_report_a_bug_properly

[luzemario]

I imagine if you force to use a downloader wich not support rsync URPMI would change mirror to one inside downloader's supported protocol. Mirrors are dinamically handled, but this "dinamically" fashion is not allways the best chioce. You have no control over your downloading, bringing you to long timeouts until URPMI "detects" a mirror (or connection to it) is failing. Here, very often clicking on "Cancel" button results in long delays and database locks when drakprm is killed.

There are several issues related to updating/package installing with Mandriva graphical tools when connection is not good. If you are a novice coming from Windows and has a unstable or slow connection, you can think your computer is frozen very often. This will make novice users go away.

I am a old bug reporter, since old Mandriva times. That bug are so old too. Unplug your network cable and try to run drakrpm programs by yourself, you will see things not present on good networks when trying to download.

Going back to the bug, here are my proxy/urpmi.cfg working setup:

urpmi.cfg

Code: Select all

{
  curl-options: --proxy-ntlm
  downloader: curl
  verify-rpm: 1
  xml-info: always
}
(...stripped)


proxy.cfg (my proxy is named "proxy" too)

Code: Select all

http_proxy=proxy:3128
ftp_proxy=proxy:3128
proxy_user_ask


Urpmi asks for proxy user and password, and instructs curl to authenticate with correct protocol and all will be fine. But it is not the best for novice user. Graphical Mageia tools must have option to allow to do it with NO command line at all. For example, "proxy_user_ask" must be a checkbox in drakproxy, for activation and desactivation inside mcc.

Wihtin this setup, urpmi works as expected. But gurpmi does not. Despite you fill in correct proxy data, you get HTTP 407 errors.

[luzemario]

Created bug for urpmi ignoring "--proxy-user=ask" parameter: https://bugs.mageia.org/show_bug.cgi?id=9870

[doktor5000]

I imagine if you force to use a downloader wich not support rsync URPMI would change mirror to one inside downloader's supported protocol.

Mirrors are only dynamically chosen if you use $MIRRORLIST (which is the default). If you prefer one mirror or protocol, just add that one directly, or maybe even multiple mirrors by using

Code: Select all

urpmi --distrib URLOFMIRROR

[luzemario]

Mirrors are only dynamically chosen if you use $MIRRORLIST (which is the default). If you prefer one mirror or protocol, just add that one directly, or maybe even multiple mirrors by using

Code: Select all

urpmi --distrib URLOFMIRROR

Yeah, I know. But I allways see the whole thing as new user do. I believe Mageia is the more user-friendly distro already seen, but small things make it disappointing. I hear things like "it can be done out of the box", "why this resource is not in graphical interface", etc... Mageia has powerful scripts to automate tasks, but the same scripts misses important needs in today use.

I think Mageia can be mass-adopted with small modifications, by a lot of people. But this people only need to know Windows fundamentals to start. Mageia does not need to be Windows-like, but it can do things with less command line than today.

N.B.: Iam not against command line. The most things I do use it. But I like to see my friend doing it only by MCC too.

[doktor5000]

Configuring complex proxy authentication for outbound connections is not a task for novice users. Apart from that,
i can't comprehend what you mean. Mageia has been mass-adopted, and i don't know what you're referring to in particular.

[luzemario]

Mageia "mass-adoption" can be country-specific... here in Brazil WINDOWS is mass-adopted... I can securely say Mageia is unknown by most of people here... less than 0,1% of brazilian PCs use Linux, and yet lesser use Mageia. I mean Mageia is cool enough to be mass-adopted by millions in my country, not thousands.

And, why not proxy config can't be made easy? If it work in Windows in a easy way, surely Mageia can do too.

The "complex" part is adding a field in gurpmi labeled "Additional configurations to pass to urpmi/downloader". It can solve all trouble for specific scenarios.

[doktor5000]

The "complex" part is adding a field in gurpmi labeled "Additional configurations to pass to urpmi/downloader". It can solve all trouble for specific scenarios.

Feel free to add a patch for that to the bug report. And as you can see from the comments of the OP, just passing the addtional options to the downloader doesn't solve the problem.

For the mass adoption, i was clearly referring to those users who are already using linux, or are experimenting with it (in your case among the 0.1% linux users)
You will not get masses of people switch to linux from windows. But feel free to give some realistic proposals, what could be done
to spread Mageia even further in your country, and use a separate thread for that.

[luzemario]

Doktor,

Thanks for your time. I realize I can appear to be angry, but believe me, I am not. I want to help, but I don't know how Mageia scripts work. I know they are Python scripts and modules, but I could not figure out how to make changes without getting out of Mageia standards.

Please, can you point me some thread or wiki about Mageia tools tweaking? I want to propose patches, but I don't want to make things on-the-fly.

[luzemario]

Proxy authentication and downloaders

I will discuss here my findings about downloading Mageia updates behind a authenticated proxy connection. I will treat Rsync here as a downloader (for updating with Mageia it is downloading at all, isn't?). For clarifying purposes, when I say "behind a proxy", I mean a proxy wich needs authentication to traverse.

- WGET

Wget is a awesome downloader, full of optiions to help resuming, speed up and manage control. But for downloading behind a proxy, wget only authenticates using 'BASIC' scheme. BASIC scheme is authenticating with plain-text only, so expect it to not work inside great enterprises. Most IT admins will not allow plain-text passwords sneaking into their network for security reasons.

- ARIA2

Aria is one of the most powerful downloaders today, and has options for scripting, metalinking, bittorrent, lists and everithing more a programmer can dream. But behind a proxy, aria2c can authenticate using only 'BASIC' and 'DIGEST' schemes. Some Windows IT adminis can restrict proxy authentication to NTLM (scheme used in Windows networks), rendering aria2c unusable.

- RSYNC

Rsync is a must-have for anyone thinking about bandwidth. It has advanced options for resuming, bandwidth control, diff blocks and conditional copying, making it the best for busy file servers and mirrors in general. Rsync was not designed to work behind a proxy, so it has no options to help to traverse or authenticating with one. The only way to use rsync behind a proxy is to direct it by a authenticated VPN or tunnel.

- CURL

Curl may be the only choice to traverse proxies based in Windows-only networks. It has the authentication schemes 'BASIC', 'DIGEST' and 'NTLM'. Curl can be instructed to use one or all, or to try the best one automatically. Curl has embedded Kerberos support, so it may be used without help of the host system to authenticate in more restrictive proxy setups.

All above apply only to authentication of proxy to get outside internal network. if you use a transparent proxy, or a proxy with caching only function all downloaders will probably do work. If your intranet is of type wich resolves outside DNS and allow pings to external sites but do not allow unauthenticated TCP connections, you will likely see GURPMI displaying a 'wait' screen with no progress forever.