[Solved] What is the proper place for an unmount service?

[jiml8]

Yesterday, I shut my workstation down to make some hardware changes, but I totally forgot to dismount my NAS before shutting down.

This apparently caused the shutdown to hang, and - of course - I did not have any consoles to debug exactly why the shutdown was hung. I was very tired, and wanted to get on with it, so I punched the power button on the workstation. My hardware changes (actually, a firmware change and fix for my SSDs) took about 3 hours, and during that time another computer was streaming videos from that NAS.

When I brought my workstation back up, and tried to connect to the NAS, the NAS crashed and wound up with a substantially damaged filesystem. In fact, I had to do a partial restore from backup.

I have observed in the past that rebooting this workstation without dismounting from the NAS can cause the NAS problems, but this time the fact that I was using the NAS from another computer after it got tangled up by losing my workstation made things bad.

Normally, the workstation has an iscsi share mounted from the NAS, and an NFS share, and a webdav share. I think the iscsi share is the one that tangles things up, but I am not certain of that. The other computer that was accessing the NAS was using CIFS to access the same NAS directory that the workstation mounts using NFS.

So, to deal with the fact that I often forget to dismount the NAS, I wrote a systemd service to handle all of that for me.

Here is the service, which I located in /etc/systemd/system):

Code: Select all

[Unit]
Description = Unmount jssnas before shutdown
Before = final.target
After = shutdown.target
DefaultDependencies = no

[Service]
ExecStart = /usr/bin/unmountNAS

[Install]
WantedBy = shutdown.target


and here is the script unmountNAS that I wrote. Because I often have sshfs mounts running as well, I decided this would be a good place to unmount those too:

Code: Select all

#! /bin/bash

# Unmount the JSS NAS and also any fuser filesystems that are mounted
# which is likely to include a webdav mount
# After all, we are shutting down so should clean things up.
#
umount /mnt/NAS
iscsiadm -m node --logout all
umount -f /mnt/Video
toggle=0
for fuse in fuse fuse.sshfs
do
  Mnts=`mount -t $fuse`;
  if [ -n "$Mnts" ]; then
    for Mnt in $Mnts
    do                                                                               
      if [ "$Mnt" == "on" ]; then                                                                 
        toggle=1;                                                                                 
      elif [ "$toggle" -eq 1 ]; then                                                               
        toggle=0;                                                                                 
        fusermount -u $Mnt;                                                                       
      fi                                                                                           
    done                                                                                           
  fi                                                                                               
done

So, OK. I enabled my unmountnas.service, and this all works fine. I can unmount all this stuff by executing "systemctl start unmountnas.service".

My problem is that I am not quite sure where I want this located in the shutdown sequence. I am pretty sure that where I have placed it is too late in the shutdown sequence. It needs to run before stopping the network.

However, I am still wrapping my head around this systemd stuff, and I don't have a good handle about how the shutdown actually proceeds. Also, given that I can wind up with a damaged NAS if I get it wrong (hence, debugging is painful), I decided to ask here.

What should I put in the service definition for the Before and After parameters? Also, is my WantedBy correct?

[doktor5000]

I've asked our systemd maintainer to have a look, but he's usually quite busy ...

Maybe it helps if you have a look at shutdown.target or especially

umount.target
A special target unit that umounts all mount and automount points on system shutdown.

Mounts that shall be unmounted on system shutdown shall add Conflicts dependencies to this unit for their mount unit, which is implicitly done when DefaultDependencies=yes is set
(the default).

[jiml8]

I will look at that, but if it is symmetrical with how mounts occur on startup, it will be in the wrong place. In fact, I suspect it is the source of the hang on shutdown as it tries to dismount network shares after the network is closed.

On startup, systemd by default tries to mount the NAS shares at the wrong time (too early), fails, and hangs. It tells me it is going to a repair console but never gets there, forcing me to boot into a repair distro. I have dealt with this by making the mounts "noauto" in fstab, and constructing a mountnas.service that runs pretty much at the end of the boot sequence (Wants = display-manager.service, After = NetworkManager-wait-online.service network.target, WantedBy = graphical.target).

I want to dismount the NAS the same way. I want to dismount it as soon as the user session is closed and prior to the network being stopped.

I have changed the unmountnas.service to "Before = shutdown.target, WantedBy = exit.target" which, based on the manpage you pointed out to me, should be much earlier in the shutdown sequence. I think. At least, I presume that exit.target (which kills the user manager) only runs at shutdown...

[doktor5000]

Usually you should not mark them noauto in fstab but as _netdev to mark them as remote mounts. Systemd should adjust the ordering accordingly.
https://wiki.mageia.org/en/Mageia_4_Err ... rrect_UUID

[jiml8]

The emphasis is on "systemd should adjust...". It doesn't. The iscsi share is marked as _netdev in fstab. Doesn't matter.

I don't think _netdev is available as an option for the NFS mount (IIRC I tried it and got an error), but it is NFS which means it is remote.

[jiml8]

I have changed the unmountnas.service to "Before = shutdown.target, WantedBy = exit.target".

As it happens, I had to reboot yesterday because the last of the old hard drives in my system - a 4.5 year old WD Green drive - was experiencing rapid growth of its bad blocks list. So I replaced it.

On shutdown, all the network mounts that I wanted this service to close were properly closed, and before networking shut down. Just the way I need it to happen.

I had ordered the machine to shut down, however, and it didn't. Or, rather, it only partly shut down. Display went black, lights on mobo (other than the 5V power light) went out. I believe, but am not certain, that the hard drives stopped spinning.

But the fans continued to run. So, I don't know exactly how far down the system really was. I did kill the fans (and whatever else was still running) with the power button.

And, of course, when I restarted the system with a brand new blank HD in place, the system refused to boot because I had forgotten to put "nofail" in the fstab entry for the drive that was removed. When it failed to boot, naturally it also failed to actually get to a repair console. It merely told me it was going to drop to a repair console, then hung. This required me to boot from a DVD in order to fix the problem.

This is the systemd "better way" at work, yet again.