[Solved] Mounting network drives causing long boot

This forum is dedicated to basic help and support :

Ask here your questions about basic installation and usage of Mageia. For example you may post here all your questions about getting Mageia isos and installing it, configuring your printer, using your word processor etc.

Try to ask your questions in the right sub-forum with as much details as you can gather. the more precise the question will be, the more likely you are to get a useful answer

[Solved] Mounting network drives causing long boot

Postby mark9117 » Jan 25th, '18, 23:43

Mageia release 6 (Official) for x86_64
It's a mature install, not sure exactly how old, but this issue has existed since it was first installed.

What's happening is that I am experiencing about a 2 minute delay when booting the machine. I'm watching standard out and seeing that network drives are showing mounted, but they aren't. Then I see a job start running to mount drives:

Image

That image is cutting off the right-hand portion for some reason. The timeout on that job is 1:56 - almost 2 minutes. You can see that the job is 53 seconds in at the point that image was captured. The job simply times out and the drives have to be mounted manually (sudo mount -a) after boot completes.

My fstab has all of these drives configured like this:

Code: Select all
adamsmdk:/backup-1 /mnt/adamsmdk/backup-1 nfs nosuid,rsize=8192,wsize=8192,_netdev 0 0


<server>:<share> <mount-point> nfs nosuid,rsize=8192,wsize=8192,_netdev 0 0

This shows up like this under the "mount" command:

Code: Select all
adamsmdk:/backup-1 on /mnt/adamsmdk/backup-1 type nfs4 (rw,nosuid,relatime,vers=4.2,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.106,local_lock=none,addr=192.168.1.100,_netdev)


The shares are all on nfs servers and conform to this configuration in /etc/exports:

Code: Select all
# generated by drakhosts.pl
/backup-1 *(no_all_squash,async,secure,no_subtree_check,rw)


The "_netdev" option doesn't really seem to have any effect on the process, but it doesn't hurt anything either.

Systemd-analyze blame says

Code: Select all
# systemd-analyze blame
          9.881s network-up.service
          5.057s webmin.service
          4.163s mandriva-everytime.service
          4.035s systemd-udev-settle.service
          3.514s shorewall6.service
          2.962s mysqld.service
          2.267s dev-sda6.device
          2.023s dev-sda1.device
          1.822s systemd-journal-flush.service
          1.547s systemd-fsck@dev-disk-by\x2duuid-dcac9b29\x2dee71\x2d457f\x2d9750\x2dec513ad35610.service
          1.430s systemd-logind.service
          1.420s cpupower.service
          1.419s gpm.service
          1.417s rpcbind.service
          1.405s rsyslog.service
          1.391s gssproxy.service
          1.391s partmon.service
          1.390s iptables.service
          1.389s ipmievd.service
          1.389s mga-bg-res.service
          1.389s acpid.service
          1.301s systemd-fsck@dev-disk-by\x2duuid-bf27abc3\x2d578f\x2d457c\x2da26f\x2d41b36a8299b1.service
          1.296s systemd-fsck@dev-disk-by\x2duuid-8e95887b\x2d58ac\x2d4d78\x2da89b\x2d2dcba7f41916.service
          1.231s systemd-fsck@dev-disk-by\x2duuid-6c736e4e\x2d9257\x2d44b0\x2da1a1\x2dfb133086d3df.service
          1.150s avahi-daemon.service
          1.131s bluetooth.service
          1.075s systemd-fsck@dev-disk-by\x2duuid-def9da8b\x2d5dff\x2d4ed3\x2dbeca\x2d6e6e64875dc3.service
          1.073s systemd-fsck@dev-disk-by\x2duuid-7a73b4c4\x2d7dc6\x2d42a7\x2db249\x2ddc2e14df1e66.service
           995ms httpd.service
           983ms cups.service
           918ms media-windows.mount
           911ms systemd-fsck@dev-disk-by\x2duuid-5cc5a587\x2df5ed\x2d4a9a\x2d9d78\x2df5356bd5a6fe.service
           824ms resolvconf.service
           812ms fedora-storage-init.service
           791ms network.service
           720ms mnt-backup.mount
           716ms systemd-fsck@dev-disk-by\x2duuid-7aa6f935\x2d8a3e\x2d4e87\x2d8357\x2db5c87b146b3e.service
           641ms sensord.service                                                                                                                                             
           631ms home.mount                                                                                                                                                 
           604ms lm_sensors.service                                                                                                                                         
           564ms sshd.service                                                                                                                                               
           563ms fedora-loadmodules.service                                                                                                                                 
           547ms systemd-fsck@dev-disk-by\x2duuid-f601ac26\x2da9df\x2d4627\x2dbb7a\x2d82ecee9c6bba.service                                                                   
           472ms usr-local.mount                                                                                                                                             
           428ms systemd-backlight@backlight:acpi_video0.service                                                                                                             
           428ms systemd-random-seed.service                                                                                                                                 
           416ms systemd-tmpfiles-setup.service                                                                                                                             
           408ms systemd-vconsole-setup.service                                                                                                                             
           404ms dev-disk-by\x2duuid-672d47e9\x2d0d1e\x2d4411\x2da897\x2db61b8adb51df.swap                                                                                   
          9.881s network-up.service
          5.057s webmin.service
          4.163s mandriva-everytime.service
          4.035s systemd-udev-settle.service
          3.514s shorewall6.service
          2.962s mysqld.service
          2.267s dev-sda6.device
          2.023s dev-sda1.device
          1.822s systemd-journal-flush.service
          1.547s systemd-fsck@dev-disk-by\x2duuid-dcac9b29\x2dee71\x2d457f\x2d9750\x2dec513ad35610.service
          1.430s systemd-logind.service
          1.420s cpupower.service
          1.419s gpm.service
          1.417s rpcbind.service
          1.405s rsyslog.service
          1.391s gssproxy.service
          1.391s partmon.service
          1.390s iptables.service
          1.389s ipmievd.service
          1.389s mga-bg-res.service
          1.389s acpid.service
          1.301s systemd-fsck@dev-disk-by\x2duuid-bf27abc3\x2d578f\x2d457c\x2da26f\x2d41b36a8299b1.service
          1.296s systemd-fsck@dev-disk-by\x2duuid-8e95887b\x2d58ac\x2d4d78\x2da89b\x2d2dcba7f41916.service
          1.231s systemd-fsck@dev-disk-by\x2duuid-6c736e4e\x2d9257\x2d44b0\x2da1a1\x2dfb133086d3df.service
          1.150s avahi-daemon.service
          1.131s bluetooth.service
          1.075s systemd-fsck@dev-disk-by\x2duuid-def9da8b\x2d5dff\x2d4ed3\x2dbeca\x2d6e6e64875dc3.service
          1.073s systemd-fsck@dev-disk-by\x2duuid-7a73b4c4\x2d7dc6\x2d42a7\x2db249\x2ddc2e14df1e66.service
           995ms httpd.service
           983ms cups.service
           918ms media-windows.mount
           911ms systemd-fsck@dev-disk-by\x2duuid-5cc5a587\x2df5ed\x2d4a9a\x2d9d78\x2df5356bd5a6fe.service
           824ms resolvconf.service
           812ms fedora-storage-init.service
           791ms network.service
           720ms mnt-backup.mount
           716ms systemd-fsck@dev-disk-by\x2duuid-7aa6f935\x2d8a3e\x2d4e87\x2d8357\x2db5c87b146b3e.service
           641ms sensord.service
           631ms home.mount
           604ms lm_sensors.service
           564ms sshd.service
           563ms fedora-loadmodules.service
           547ms systemd-fsck@dev-disk-by\x2duuid-f601ac26\x2da9df\x2d4627\x2dbb7a\x2d82ecee9c6bba.service
           472ms usr-local.mount
           428ms systemd-backlight@backlight:acpi_video0.service
           428ms systemd-random-seed.service
           416ms systemd-tmpfiles-setup.service
           408ms systemd-vconsole-setup.service
           404ms dev-disk-by\x2duuid-672d47e9\x2d0d1e\x2d4411\x2da897\x2db61b8adb51df.swap
          9.881s network-up.service
          5.057s webmin.service
          4.163s mandriva-everytime.service
          4.035s systemd-udev-settle.service
          3.514s shorewall6.service
          2.962s mysqld.service
          2.267s dev-sda6.device
          2.023s dev-sda1.device
          1.822s systemd-journal-flush.service
          1.547s systemd-fsck@dev-disk-by\x2duuid-dcac9b29\x2dee71\x2d457f\x2d9750\x2dec513ad35610.service
          1.430s systemd-logind.service
          1.420s cpupower.service
          1.419s gpm.service
          1.417s rpcbind.service
          1.405s rsyslog.service
          1.391s gssproxy.service
          1.391s partmon.service
          1.390s iptables.service
          1.389s ipmievd.service
          1.389s mga-bg-res.service
          1.389s acpid.service
          1.301s systemd-fsck@dev-disk-by\x2duuid-bf27abc3\x2d578f\x2d457c\x2da26f\x2d41b36a8299b1.service
          1.296s systemd-fsck@dev-disk-by\x2duuid-8e95887b\x2d58ac\x2d4d78\x2da89b\x2d2dcba7f41916.service
          1.231s systemd-fsck@dev-disk-by\x2duuid-6c736e4e\x2d9257\x2d44b0\x2da1a1\x2dfb133086d3df.service
          1.150s avahi-daemon.service
          1.131s bluetooth.service
          1.075s systemd-fsck@dev-disk-by\x2duuid-def9da8b\x2d5dff\x2d4ed3\x2dbeca\x2d6e6e64875dc3.service
          1.073s systemd-fsck@dev-disk-by\x2duuid-7a73b4c4\x2d7dc6\x2d42a7\x2db249\x2ddc2e14df1e66.service
           995ms httpd.service
           983ms cups.service
           918ms media-windows.mount
           911ms systemd-fsck@dev-disk-by\x2duuid-5cc5a587\x2df5ed\x2d4a9a\x2d9d78\x2df5356bd5a6fe.service
           824ms resolvconf.service
           812ms fedora-storage-init.service
           791ms network.service
           720ms mnt-backup.mount
           716ms systemd-fsck@dev-disk-by\x2duuid-7aa6f935\x2d8a3e\x2d4e87\x2d8357\x2db5c87b146b3e.service
           641ms sensord.service
           631ms home.mount
           604ms lm_sensors.service
           564ms sshd.service
           563ms fedora-loadmodules.service
           547ms systemd-fsck@dev-disk-by\x2duuid-f601ac26\x2da9df\x2d4627\x2dbb7a\x2d82ecee9c6bba.service
           472ms usr-local.mount
           428ms systemd-backlight@backlight:acpi_video0.service
           428ms systemd-random-seed.service
           416ms systemd-tmpfiles-setup.service
           408ms systemd-vconsole-setup.service
           404ms dev-disk-by\x2duuid-672d47e9\x2d0d1e\x2d4411\x2da897\x2db61b8adb51df.swap
           393ms upower.service                                                                                                                                             
           390ms fedora-readonly.service                                                                                                                                     
           375ms systemd-fsck-root.service
           333ms mandriva-save-dmesg.service


Once the shares are mounted by root with "mount -a" everything works just fine, but how do I resolve the trouble the system is having on boot? I'm starting to think I need to just go with automount, which works well on my laptops, but this is a desktop that should work better than it does.

Any ideas?
Last edited by mark9117 on Apr 1st, '18, 20:57, edited 1 time in total.
Let's just reboot everything all the time.
User avatar
mark9117
 
Posts: 395
Joined: Sep 12th, '11, 20:32
Location: Eastern New Mexico -- Not Hell, but you can see it from here.

Re: Mounting network drives causing long boot

Postby JoesCat » Jan 28th, '18, 00:52

I was reading your post (for ideas, my problem is likely a UEFI vs BIOS-legacy issue), but conclude/agree that you are definitely trying to mount before the network is available.

Looking at ideas, there is maybe "x-systemd.automount" seen here (plus an rc delay hack):
https://askubuntu.com/questions/399643/ ... ng-at-boot

...there seem to be plenty of bug reports about _netdev, this one looks interesting.
before trying the hack mentioned in the link above, look at this:
https://bugs.contribs.org/show_bug.cgi?id=8590
User avatar
JoesCat
 
Posts: 177
Joined: Sep 15th, '11, 04:27
Location: Richmond, BC, Canada

Re: Mounting network drives causing long boot

Postby wintpe » Jan 29th, '18, 17:04

its not this is it

used to cause me issues

viewtopic.php?f=8&t=6980&p=44936#p44936

regards peter
Redhat 6 Certified Engineer (RHCE)
Sometimes my posts will sound short, or snappy, however its realy not my intention to offend, so accept my apologies in advance.
wintpe
 
Posts: 1204
Joined: May 22nd, '11, 17:08
Location: Rayleigh,, Essex , UK

Re: Mounting network drives causing long boot

Postby mark9117 » Mar 24th, '18, 06:29

I had a chance to work on this tonight and found something that may or may not have any bearing on the issue.

I found this in the journalctl log:

Code: Select all
Mar 23 22:18:22 spike systemd[1]: mnt-pvr-fox.mount: Mounting timed out. Stopping.
Mar 23 22:18:22 spike systemd[1]: Mounted /mnt/pvr/fox.
Mar 23 22:18:22 spike systemd[1]: mnt-pvr-expo.mount: Mounting timed out. Stopping.
Mar 23 22:18:22 spike systemd[1]: mnt-adamsmdk-madams.mount: Mounting timed out. Stopping.
Mar 23 22:18:22 spike systemd[1]: Mounted /mnt/adamsmdk/madams.
Mar 23 22:18:22 spike systemd[1]: Mounted /mnt/pvr/expo.
Mar 23 22:18:22 spike systemd[1]: mnt-pvr-burbank.mount: Mounting timed out. Stopping.
Mar 23 22:18:22 spike systemd[1]: Mounted /mnt/pvr/burbank.
Mar 23 22:18:22 spike systemd[1]: mnt-pvr-cineplex.mount: Mounting timed out. Stopping.
Mar 23 22:18:22 spike systemd[1]: Mounted /mnt/pvr/cineplex.
Mar 23 22:18:22 spike systemd[1]: mnt-shuttle-madams.mount: Mounting timed out. Stopping.
Mar 23 22:18:22 spike systemd[1]: Mounted /mnt/shuttle/madams.
Mar 23 22:18:22 spike systemd[1]: mnt-shuttle-www.mount: Mounting timed out. Stopping.
Mar 23 22:18:22 spike mount[1769]: mount.nfs: Unknown error 512
Mar 23 22:18:22 spike mount[1786]: mount.nfs: Unknown error 512
Mar 23 22:18:22 spike mount[1758]: mount.nfs: Unknown error 512
Mar 23 22:18:22 spike mount[1777]: mount.nfs: Unknown error 512
Mar 23 22:18:22 spike systemd[1]: Mounted /mnt/shuttle/www.
Mar 23 22:18:22 spike systemd[1]: mnt-pvr-madams.mount: Mounting timed out. Stopping.
Mar 23 22:18:22 spike systemd[1]: Mounted /mnt/pvr/madams.
Mar 23 22:18:22 spike systemd[1]: mnt-pvr-tvfiles.mount: Mount process exited, code=exited status=32
Mar 23 22:18:22 spike systemd[1]: Failed to mount /mnt/pvr/tvfiles.
Mar 23 22:18:22 spike systemd[1]: Dependency failed for Remote File Systems.
Mar 23 22:18:22 spike systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
Mar 23 22:18:22 spike systemd[1]: mnt-pvr-tvfiles.mount: Unit entered failed state.
Mar 23 22:18:22 spike systemd[1]: mnt-pvr-downtown.mount: Mount process exited, code=exited status=32


I started tracking that down and got nowhere pretty fast. It seems error 512 is a generic error dealing withAUTH_GSS upcall failing. I poked around that information and found that rpc-svcgssd is not running on the servers that are failing some of the shares (oddly, some shares on those servers are mounting but not consistently).

The rpc-svcgssd service is set to start at boot, but shows that it is not running after a reboot and I can't get it started manually:

Code: Select all
# systemctl status rpc-svcgssd
● rpc-svcgssd.service - RPC security service for NFS server
   Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static; vendor preset: enabled)
   Active: inactive (dead)
Condition: start condition failed at Fri 2018-03-23 21:49:49 MDT; 25s ago
           ConditionPathExists=/etc/krb5.keytab was not met


I looked into exports and found some information about the "security" option in the nfs export config. I changed that to insecure (nfs port was 2049) and it did not resolve the issue.

For the record, still stumped.
Let's just reboot everything all the time.
User avatar
mark9117
 
Posts: 395
Joined: Sep 12th, '11, 20:32
Location: Eastern New Mexico -- Not Hell, but you can see it from here.

Re: Mounting network drives causing long boot

Postby doktor5000 » Mar 24th, '18, 17:03

mark9117 wrote:I found this in the journalctl log:

You're missing context information, you found that in the log but what was happening at the time those messages were logged? Was that during boot, after boot, ... ?

Regarding your question why the service did not start, the journal output clearly says why:
mark9117 wrote:
Code: Select all
# systemctl status rpc-svcgssd
Condition: start condition failed at Fri 2018-03-23 21:49:49 MDT; 25s ago
           ConditionPathExists=/etc/krb5.keytab was not met



As you don't use kerberos and hence don't have a keytab on the NFS server that service won't start, so no point in trying to start it.

Also see https://superuser.com/questions/1201159 ... 46#1207346 and maybe https://bugzilla.redhat.com/show_bug.cgi?id=1428941
which provides basically the same information as Peter did with the link to his thread - did you actually read that yet?
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17629
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: Mounting network drives causing long boot

Postby jiml8 » Mar 24th, '18, 17:36

I suggest you remove the _netdev option and replace it with the nofail option on your nfs mounts. Then the system won't wait for them to mount...which can't happen until networking is started anyway, and at a guess I would say you probably are trying to mount these nfs devices before networking is up.
jiml8
 
Posts: 1253
Joined: Jul 7th, '13, 18:09

Re: Mounting network drives causing long boot

Postby mark9117 » Mar 26th, '18, 02:56

doktor5000 wrote:
mark9117 wrote:I found this in the journalctl log:

You're missing context information, you found that in the log but what was happening at the time those messages were logged? Was that during boot, after boot, ... ?


Of course you're right. There is no context associated with that blurb. Here is the log from my most recent boot. The failure of the nfs shares to mount shows up around line 1476.

This boot occurred after I blacklisted the kerberos module via the command:

Code: Select all
echo "blacklist rpcsec_gss_krb5" > /etc/modprobe.d/blacklist-nfs-gss-krb5.conf


The remote shares did not mount.
Let's just reboot everything all the time.
User avatar
mark9117
 
Posts: 395
Joined: Sep 12th, '11, 20:32
Location: Eastern New Mexico -- Not Hell, but you can see it from here.

Re: Mounting network drives causing long boot

Postby mark9117 » Mar 26th, '18, 03:02

jiml8 wrote:I suggest you remove the _netdev option and replace it with the nofail option on your nfs mounts. Then the system won't wait for them to mount...which can't happen until networking is started anyway, and at a guess I would say you probably are trying to mount these nfs devices before networking is up.


Thanks for the reply Jim.

I replaced the _netdev option with nofail on some of the shares:

Code: Select all
### nofail
adamsmdk:/backup-1 /mnt/adamsmdk/backup-1 nfs nosuid,rsize=8192,wsize=8192,nofail 0 0
adamsmdk:/mnt/backup-2 /mnt/adamsmdk/backup-2 nfs wsize=8192,rsize=8192,nosuid,nofail 0 0
adamsmdk:/home/madams /mnt/adamsmdk/madams nfs nosuid,rsize=8192,wsize=8192,nofail  0 0
adamsmdk:/mnt/well /mnt/adamsmdk/well nfs nosuid,wsize=8192,rsize=8192,nofail 0 0
### _netdev
pvr:/mnt/burbank /mnt/pvr/burbank nfs nosuid,rsize=8192,wsize=8192,_netdev 0 0
pvr:/mnt/cineplex /mnt/pvr/cineplex nfs nosuid,wsize=8192,rsize=8192,_netdev 0 0
pvr:/mnt/downtown /mnt/pvr/downtown nfs rsize=8192,wsize=8192,nosuid,_netdev, 0 0
pvr:/mnt/expo /mnt/pvr/expo nfs wsize=8192,rsize=8192,nosuid,_netdev 0 0
pvr:/mnt/fox /mnt/pvr/fox nfs rsize=8192,wsize=8192,nosuid,_netdev 0 0
pvr:/mnt/imax /mnt/pvr/imax nfs nosuid,rsize=8192,wsize=8192,_netdev 0 0
pvr:/home/madams /mnt/pvr/madams nfs nosuid,wsize=8192,rsize=8192,_netdev 0 0
pvr:/mnt/myth /mnt/pvr/myth nfs nosuid,rsize=8192,wsize=8192,_netdev 0 0
pvr:/mnt/tvfiles /mnt/pvr/tvfiles nfs rsize=8192,wsize=8192,nosuid,_netdev 0 0


Regrettably, this did not make any difference after several boots. I would also note that blacklisting the kerberos module made no difference.

Also, see the log from my last boot. https://pastebin.com/U6mWGv8M

It looks to me as if the network comes up around line 1248. That's well before the system starts trying to mount the nfs shares at line 1308, and before the mounts fail at around 1476.
Let's just reboot everything all the time.
User avatar
mark9117
 
Posts: 395
Joined: Sep 12th, '11, 20:32
Location: Eastern New Mexico -- Not Hell, but you can see it from here.

Re: Mounting network drives causing long boot

Postby jiml8 » Mar 26th, '18, 16:54

If you only tried the change on some nfs connections, then I would not expect you to see a difference; the system will still pause while an attempt is made to connect the other ones.

I note that you are mounting these shares by a uri name. Maybe you have a DNS problem and those names are not being resolved?

I mount one NFS share on boot. It always mounts without any issue, but I do mount it by IP. In case it matters, here is the fstab entry:
Code: Select all
192.168.2.27:/mnt/VD01/Videos /mnt/Video nfs rsize=32768,wsize=32768,soft,nosuid,noacl,noauto,nofail
jiml8
 
Posts: 1253
Joined: Jul 7th, '13, 18:09

Re: Mounting network drives causing long boot

Postby mark9117 » Mar 27th, '18, 06:55

jiml8 wrote:If you only tried the change on some nfs connections, then I would not expect you to see a difference; the system will still pause while an attempt is made to connect the other ones.


Agreed, but the system showed that it was trying to mount all the shares just as it always does. And it failed to mount them all, just as it does most of the time. On some reboots, it will mount shares from the "pvr" server, but it isn't consistent or reliable in any way.

I note that you are mounting these shares by a uri name. Maybe you have a DNS problem and those names are not being resolved?

I mount one NFS share on boot. It always mounts without any issue, but I do mount it by IP. In case it matters, here is the fstab entry:
Code: Select all
192.168.2.27:/mnt/VD01/Videos /mnt/Video nfs rsize=32768,wsize=32768,soft,nosuid,noacl,noauto,nofail


I'll give those options a try as soon as I get the chance.
Let's just reboot everything all the time.
User avatar
mark9117
 
Posts: 395
Joined: Sep 12th, '11, 20:32
Location: Eastern New Mexico -- Not Hell, but you can see it from here.

Re: Mounting network drives causing long boot

Postby hviaene » Mar 27th, '18, 12:36

I didn't check on the time for booting, but I noticed also that my nfs shares were not mounted, needing a mont - a after each boot. I found in the journal that the FQDN of the nfs shares could nit be resolved at boot, or in other words: the mounting of the nfs shares occurs before the network (Wif in my case) was fully initiated.
hviaene
 
Posts: 143
Joined: Oct 11th, '13, 10:41

Re: Mounting network drives causing long boot

Postby jiml8 » Mar 27th, '18, 19:34

If all the nfs shares are set to nofail, then of course the system will try to mount them at startup, but it will not wait for them to mount. So your boot will proceed quickly, and those shares will actually get mounted when enough of the system is running for that to happen.
jiml8
 
Posts: 1253
Joined: Jul 7th, '13, 18:09

Re: Mounting network drives causing long boot

Postby mark9117 » Mar 30th, '18, 01:05

First of all, thanks for all the responses here. I finally got some time to work on this issue.

The current state of the relevant part of my fstab file his this:

Code: Select all
adamsmdk.local:/backup-1 /mnt/adamsmdk/backup-1 nfs nosuid,rsize=8192,wsize=8192,nofail 0 0
adamsmdk.local:/mnt/backup-2 /mnt/adamsmdk/backup-2 nfs wsize=8192,rsize=8192,nosuid,nofail 0 0
adamsmdk.local:/home/madams /mnt/adamsmdk/madams nfs nosuid,rsize=8192,wsize=8192,nofail  0 0
adamsmdk.local:/mnt/well /mnt/adamsmdk/well nfs nosuid,wsize=8192,rsize=8192,nofail 0 0
##pvr:/mnt/burbank /mnt/pvr/burbank nfs x-systemd.automount,x-systemd.device-timeout=10,nosuid,rsize=8192,wsize=8192,_netdev 0 0
pvr:/mnt/burbank /mnt/pvr/burbank nfs nosuid,rsize=8192,wsize=8192,nofail 0 0
pvr:/mnt/cineplex /mnt/pvr/cineplex nfs nosuid,wsize=8192,rsize=8192,nofail 0 0
pvr:/mnt/downtown /mnt/pvr/downtown nfs rsize=8192,wsize=8192,nosuid,nofail 0 0
pvr:/mnt/expo /mnt/pvr/expo nfs wsize=8192,rsize=8192,nosuid,nofail 0 0
pvr:/mnt/fox /mnt/pvr/fox nfs rsize=8192,wsize=8192,nosuid,nofail 0 0
pvr:/mnt/imax /mnt/pvr/imax nfs nosuid,rsize=8192,wsize=8192,nofail 0 0
pvr:/home/madams /mnt/pvr/madams nfs nosuid,wsize=8192,rsize=8192,nofail 0 0
pvr:/mnt/myth /mnt/pvr/myth nfs nosuid,rsize=8192,wsize=8192,nofail 0 0
pvr:/mnt/tvfiles /mnt/pvr/tvfiles nfs rsize=8192,wsize=8192,nosuid,nofail 0 0
shuttle:/home/madams /mnt/shuttle/madams nfs nosuid,wsize=8192,rsize=8192,nofail 0 0
shuttle:/share /mnt/shuttle/server nfs nosuid,rsize=8192,wsize=8192,nofail 0 0
shuttle:/super /mnt/shuttle/super nfs wsize=8192,rsize=8192,nosuid,nofail 0 0
shuttle:/var/www/html /mnt/shuttle/www nfs rsize=8192,wsize=8192,nosuid,nofail 0 0


Note that I have attempted to identify the "adamsmdk" server by FQDN (such that it is on my local network).

Here is what I have done/changed this afternoon:

I noticed that the system seemed to be hanging while trying to mount the share adamsmdk:/backup-1 /mnt/adamsmdk/backup-1 nfs nosuid,rsize=8192,wsize=8192,nofail 0 0

I disabled (remarked out) that share and rebooted.
When it came up, th only nfs shares mounted were the shares on pvr:

Code: Select all
$ df
Filesystem                    Size  Used Avail Use% Mounted on
devtmpfs                      3.9G     0  3.9G   0% /dev
tmpfs                         3.9G   51M  3.8G   2% /dev/shm
tmpfs                         3.9G  2.4M  3.9G   1% /run
/dev/sda1                      38G  310M   36G   1% /
/dev/sda6                      24G   13G  9.7G  58% /usr
tmpfs                         3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sdb1                     2.0T   61G  1.9T   4% /mnt/photos
/dev/sdb5                     735G  224M  698G   1% /mnt/process
/dev/sda7                     8.4G  2.2G  5.9G  27% /var
/dev/sda11                     12G  553M   11G   5% /usr/local
/dev/sda8                     987M  181M  739M  20% /boot
/dev/sdc7                     8.3G  3.7G  4.7G  44% /media/windows
/dev/sdc8                      50G   28G   20G  59% /virt
/dev/sda10                    378M   27M  328M   8% /tmp
/dev/sdc1                     121G   61G   60G  51% /home
/dev/sdc5                      78G   62G   13G  83% /share
/dev/sda9                      32G  3.0G   27G  11% /opt
/dev/sdc6                      39G  1.5G   36G   4% /mnt/backup
/dev/sdc9                      71G   26G   42G  39% /mnt/mine
tmpfs                         786M   24K  786M   1% /run/user/10001
pvr:/mnt/burbank              926G   77M  879G   1% /mnt/pvr/burbank
pvr:/mnt/cineplex             724G   15G  673G   3% /mnt/pvr/cineplex
pvr:/mnt/downtown             907G   77M  861G   1% /mnt/pvr/downtown
pvr:/mnt/expo                 406G  8.4G  377G   3% /mnt/pvr/expo
pvr:/mnt/fox                  389G  215G  155G  59% /mnt/pvr/fox
pvr:/mnt/imax                 721G  297G  388G  44% /mnt/pvr/imax
pvr:/home/madams               50G  2.4G   47G   5% /mnt/pvr/madams
pvr:/mnt/myth                 7.6G   18M  7.2G   1% /mnt/pvr/myth
pvr:/mnt/tvfiles              503G   73M  478G   1% /mnt/pvr/tvfiles


This is the excerpt from journaclctl:
Code: Select all
Mar 29 14:14:15 spike network-up[976]: Waiting for network to be up[  OK  ]
Mar 29 14:14:15 spike systemd[1]: Started LSB: Wait for the hotplugged network to be up.
Mar 29 14:14:15 spike systemd[1]: Reached target Host and Network Name Lookups.
Mar 29 14:14:15 spike systemd[1]: Reached target Network.
Mar 29 14:14:15 spike systemd[1]: Reached target Network is Online.
Mar 29 14:14:15 spike systemd[1]: Starting LSB: Webmin is a remote administration tool using web-browser...
Mar 29 14:14:15 spike systemd[1]: Starting Shorewall IPv6 firewall...
Mar 29 14:14:15 spike systemd[1]: Starting The Apache HTTP Server...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/imax...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/fox...
Mar 29 14:14:15 spike systemd[1]: Starting Notify NFS peers of a restart...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/madams...
Mar 29 14:14:15 spike systemd[1]: Starting OpenSSH server daemon...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/myth...
Mar 29 14:14:15 spike systemd[1]: Starting CUPS Scheduler...
Mar 29 14:14:15 spike systemd[1]: Starting Xinetd A Powerful Replacement For Inetd...
Mar 29 14:14:15 spike systemd[1]: Starting Permit User Sessions...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/shuttle/madams...
Mar 29 14:14:15 spike systemd[1]: Starting Network Name Resolution...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/expo...
Mar 29 14:14:15 spike systemd[1]: Starting MySQL database server...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/cineplex...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/burbank...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/adamsmdk/backup-2...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/shuttle/super...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/adamsmdk/well...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/tvfiles...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/adamsmdk/madams...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/shuttle/server...
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/pvr/downtown...
Mar 29 14:14:15 spike systemd[1]: Started GNU Krell Monitors server.
Mar 29 14:14:15 spike systemd[1]: Mounting /mnt/shuttle/www...


Only the shares from the pvr server mounted.
Subsequent in the journal:

Code: Select all
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/tvfiles.
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/expo.
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/madams.
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/myth.
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/burbank.
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/cineplex.
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/fox.
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/downtown.
Mar 29 14:14:16 spike systemd[1]: Mounted /mnt/pvr/imax.


Rebooted the system from the command line and this time all nfs shares mounted okay.
I shutdown and power cycled the system. It hung trying to mount shuttle:/super /mnt/shuttle/super nfs wsize=8192,rsize=8192,nosuid,nofail 0 0.

Journalctl from that boot:

Code: Select all
Mar 29 15:12:54 spike systemd[1]: mnt-adamsmdk-madams.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/adamsmdk/madams.
Mar 29 15:12:54 spike systemd[1]: mnt-shuttle-super.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: mnt-adamsmdk-backup\x2d2.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/shuttle/super.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/adamsmdk/backup-2.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-madams.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/pvr/madams.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-fox.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/pvr/fox.
Mar 29 15:12:54 spike systemd[1]: mnt-adamsmdk-backup\x2d1.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/adamsmdk/backup-1.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-burbank.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/pvr/burbank.
Mar 29 15:12:54 spike systemd[1]: mnt-shuttle-www.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/shuttle/www.
Mar 29 15:12:54 spike systemd[1]: mnt-shuttle-server.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/shuttle/server.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-imax.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/pvr/imax.
Mar 29 15:12:54 spike systemd[1]: mnt-adamsmdk-well.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/adamsmdk/well.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-myth.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/pvr/myth.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-cineplex.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/pvr/cineplex.
Mar 29 15:12:54 spike systemd[1]: mnt-shuttle-madams.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/shuttle/madams.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-downtown.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike mount[1776]: mount.nfs: Unknown error 512
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/pvr/downtown.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-tvfiles.mount: Mount process exited, code=exited status=32
Mar 29 15:12:54 spike systemd[1]: Failed to mount /mnt/pvr/tvfiles.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-tvfiles.mount: Unit entered failed state.
Mar 29 15:12:54 spike systemd[1]: mnt-pvr-expo.mount: Mounting timed out. Stopping.
Mar 29 15:12:54 spike systemd[1]: Mounted /mnt/pvr/expo.
Mar 29 15:12:54 spike systemd[1]: Startup finished in 3.936s (kernel) + 1min 50.966s (userspace) = 1min 54.903s.
Mar 29 15:12:56 spike sudo[2382]:   madams : TTY=tty1 ; PWD=/home/madams ; USER=root ; COMMAND=/bin/mount -a
Mar 29 15:12:56 spike systemd[1]: mnt-adamsmdk-backup\x2d2.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-adamsmdk-madams.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-adamsmdk-well.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-pvr-burbank.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-pvr-cineplex.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-pvr-downtown.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-pvr-expo.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-pvr-fox.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-pvr-imax.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-pvr-madams.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-pvr-myth.mount: Unit entered failed state.
Mar 29 15:12:56 spike systemd[1]: mnt-shuttle-madams.mount: Unit entered failed state.



Power cycled the system.

All shares mounted just fine on this boot, and no job ran during boot trying to mount these shares. I suppose at this point the presented issue - long boot time - would be resolved. But what about mounting these shares?

Uncommented the fstab share for adamsmdk.local:/backup-1 /mnt/adamsmdk/backup-1 nfs nosuid,rsize=8192,wsize=8192,nofail 0 0
Manually mounted backup-1 with no trouble.

Started KDE
Updates were available, so I updated and installed kernel 4.14.20 (probably irrelevant, but I thought I'd mention it).
Rebooted to use that kernel.

Instead of rebooting, the system just shutdown KDE and landed at the command line. Okay, I just unmounted all nfs shares manually and issued the reboot command. I unmounted those shares to save time because a job runs trying to unmount those shares and it takes about 2 minutes. If I unmount them manually, that job doesn't have to run and I save 2 minutes.

On this occasion, the system hung trying to unmount /var/tmp. I manually power cycled the system.

Here's the tail from the journal:

Code: Select all
Mar 29 15:10:32 spike systemd[1]: Shutting down.
Mar 29 15:10:32 spike systemd[1]: Hardware watchdog 'iTCO_wdt', version 0
Mar 29 15:10:32 spike systemd[1]: Set hardware watchdog to 10min.
Mar 29 15:10:32 spike kernel: watchdog: watchdog0: watchdog did not stop!
Mar 29 15:10:32 spike kernel: systemd-shutdow: 22 output lines suppressed due to ratelimiting
Mar 29 15:10:32 spike systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Mar 29 15:10:32 spike ifplugd(enp3s0)[1363]: Executing '/etc/ifplugd/ifplugd.action enp3s0 down'.
Mar 29 15:10:32 spike systemd-journald[527]: Journal stopped


I manually power cycled the system.

On reboot, all shares mounted successfully.

Started KDE. I got wallpaper and a cursor with a couple of notifications that popped up in the lower-right corner.
Killed Xorg and KDE started okay this time.

More updates. Another kernel downloaded (4.14.25).

Updates all installed okay, rebooted again.

None of the shares mounted. I manually mounted all of them with no trouble.

Did some research and it was suggested that I enable the systemd-networkd-wait-online.service. I enabled and started the service then rebooted.

The journal was sprinkled with these:

Code: Select all
Mar 29 15:29:03 spike systemd-networkd-wait-online[1198]: ignoring: lo


But nfs shares did not mount despite this in the logs:

Code: Select all
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/expo.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/madams.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/imax.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/burbank.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/fox.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/tvfiles.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/adamsmdk/backup-1.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/shuttle/www.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/adamsmdk/madams.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/adamsmdk/well.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/downtown.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/shuttle/server.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/shuttle/super.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/myth.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/pvr/cineplex.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/shuttle/madams.
Mar 29 15:29:13 spike systemd[1]: Mounted /mnt/adamsmdk/backup-2.


I edited fstab show a .local domain on one set of shares in an attempt to identify an FQDN as best I could, e.g. adamsmdk.local:/backup-1 /mnt/adamsmdk/backup-1 nfs nosuid,rsize=8192,wsize=8192,nofail 0 0

Rebooted all shares mounted okay.

Shutdown and power cycled again. Booted to an error for ATA7 showing a bad crc. There is no ATA7. Power cycled the system.

Shares did not mount
Mounted them manually okay.
Rebooted.

Shares did not mount
Mounted them manually okay.
Started KDE
More updates.
Applied updates.

Last few reboots showed nothing new in the logs.

I'm going to open a new incident to try and resolve the nfs shares not mounting. Going to wait 48 hours or so to mark this one resolved because I honestly have no idea what resolved it or if the resolution will be durable.

Again, thanks for the help on this.
Let's just reboot everything all the time.
User avatar
mark9117
 
Posts: 395
Joined: Sep 12th, '11, 20:32
Location: Eastern New Mexico -- Not Hell, but you can see it from here.


Return to Basic support

Who is online

Users browsing this forum: No registered users and 1 guest