systemd freeze on readonly nfs rootfs

[syschuck]

Hi all. I'm new to the forum, but I'm a very skilled long time linux user, from the 80's. I plan to right a howto on this if I ever get it working, but right now I need a fellow guru. I've been working on building a diskless cluster where a server exports a root files system, and a cluster node PXE boots the linux kernel and mounts the root filesystem via nfs. After 4 frustrating days (should have taken 2hrs), I finally have a node booting; but it is not without problems; specifically systemd.

So here is the story; On the server, I've setup dhcpd, bind/named, tftp, drakpxe for PXE boot server (network booting usually enabled in the ROM/BIOS of the client).I recompiled a new Mageia3 kernel from source changing nfs, dhcp, nfsroot from modules to kernel code. Then end result is a bzImage that I put in the tftpboot directory. (I'll give all the hairy details to anyone that would like to here). Bottom line is this part works perfectly. For PXEboot, it only uses that bzImage. It does not use initrd! It shouldn't need to. So the boot works flawlessly.

For the diskless filesystem, I used info from Colin Guther''s website; http://colin.guthr.ie/2012/09/nfs-root- ... ntre-v2-0/
Great resource btw. I do the following;

Code: Select all

rpm --root /diskless --initdb
urpmi --root /diskless basesystem-minimal
urpmi --root /diskless kernel-server-latest locales-en nfs-utils bash-completion colorprompt openssh-server openssh-clients task-c-devel task-c++-devel


I'm going to leave out the changes to /diskless/etc/fstab passwd, shadow etc... done to the root filesystem, but these are your typical system file changes.
I do modify /diskless/
This part is KEY to my problem. In the server's /etc/exports, I have;

Code: Select all

/diskless  10.0.0.0/24(ro,no_root_squash,no_subtree_check,async,insecure)


So /diskless should boot as a readonly nfs root filesystem. For a cluster that is what I want. Here is the problem. When the cluster node BOOTS for the FIRST TIME, it starts the whole [OK] process up to the point where it says "mageia3". It follows with a complaint about autofs4, then something about (from memory and is not exact) systemd... assert (closed_fd_id() == 0) failed. systemd; src/sys/util.c 133. The next line is systemd: Freezing system.

I looked up the source code of the error and it looks like it can't close the handle on some filehandle. It's taken 4 long days to isolate this, but if I change the
/etc/exports from ro to rw:

Code: Select all

/diskless  10.0.0.0/24(rw,no_root_squash,no_subtree_check,async,insecure)


It boots! Without error! After booting that one first time, if I change it back to ro, it will now boot OK, but systemd will start barfing that same error message about assert (close ....... == 0) failed. This time it's annoying but not fatal.

My take away is that something in the boot process changes the root filesystem on first boot. Someone has to have run into this before; and I was wondering if anyone has some idea where I can look. I suspect there is a bug in systemd, but I was wondering if anyone has been able to boot mageia3 as a readonly nfs root filesystem to a login prompt?

[doktor5000]

Then please report that as a bug, after searching if it hasn't been reported yet already: https://wiki.mageia.org/en/How_to_report_a_bug_properly

[syschuck]

Thanks for the reply. It may be a bug. I can repeat it now so, I'll go ahead and make a bug report. I finally have my notes in front of me and wanted to put it in the forum correctly in case someone searches on the assertion string. What I had previously was from memory, and yeap the details are off.

This is the code I use to build the /diskless filesystem. Hopefully this will make life easier for someone else trying to do an nfsroot filesystem. Just to repeat this bug is occurs when you export the resulting /diskless filesystem as read-only.

Code: Select all

#!/bin/bash
# This program builds a mageia root filesystem for "diskless" nfs exported filesystem.
# Run as root! CBS: 2013  Public domain.                                                                                                                                                             
#
# DESTINATION=$1
DESTINATION="/diskless"
# SERVER_IP=IP4 address of DNS/NFS server hosting diskless filesystem.
SERVER_IP=10.0.0.1

if [ -d $DESTINATION ]; then
echo "Destination ${DESTINATION} Exists Proceed anyway? (Y/N) "
read A
if [ "$A" == "N" ]; then
exit
fi
else
echo "Building root filesystem for diskless nfsroot in ${DESTINATION}"
mkdir -p $DESTINATION
fi


# STAGE1  Prepare the destination
rpm --root $DESTINATION --initdb
urpmi --root $DESTINATION --auto basesystem-minimal

# Install basic commands and utilities
urpmi --root $DESTINATION --auto kernel-server-latest locales-en nfs-utils unfs3 bash-completion colorprompt openssh-clients emacs-nox netkit-telnet-server netkit-telnet rsh rsh-server task-c-devel task-c++-devel openssh-server

# Install delevopers code mpi etc.. here.
# urpmi                                                                                                                                                                 

# STAGE2  Things in /etc
#
echo "nameserver" $SERVER_IP >> $DESTINATION/etc/resolvconf/resolv.conf.d/base
touch $DESTINATION/var/lib/random-seed
echo "node00-0" > $DESTINATION/etc/hostname
cp -f /etc/hosts /etc/passwd /etc/shadow /etc/group /etc/gshadow $DESTINATION/etc
echo "NETWORKING=yes" >> $DESTINATION/etc/sysconfig/network
echo "NEED_IDMAP=yes" >> $DESTINATION/etc/sysconfig/nfs-common
echo "READONLY=yes" > $DESTINATION/etc/sysconfig/readonly-root
#                                                                                                                                                                       
# Modify rwtab                                                                                                                                                           
# Note: This is important for a readonly root filesystem.                                                                                                               
#                                                                                                                                                                       
sed -i '/\/var\/lib\/dbus/d' $DESTINATION/etc/rwtab
echo "empty   /root
files   /var/run
files   /var/lock
files   /var/lib/dbus" > $DESTINATION/etc/rwtab.d/custom
#
# This is /etc/fstab for the nfsroot file system. You may want to edit this as you see fit.
#

echo "# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
/dev/nfs        /               nfs     sync,hard,intr,ro,nolock,rsize=8192,wsize=8192  0       0
none            /tmp            tmpfs   defaults,rw,noatime,mode=1777,nosuid,size=512M  0       0
none            /var/run        tmpfs   defaults        0       0
none            /var/lock       tmpfs   defaults        0       0
none            /var/tmp        tmpfs   defaults,rw,noatime,mode=1777,nosuid,size=512M  0       0
none            /root           tmpfs   defaults,rw,noatime,mode=1777,nosuid,size=512M  0       0
${SERVER_IP}:/home  /home           nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192  0       0
${SERVER_IP}:/usr/local /usr/local  nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192  0       0
# 10.0.0.1:/raid0  /raid0               nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192  0       0
# 10.0.0.1:/raid1  /raid1               nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192  0       0
" > $DESTINATION/etc/fstab
 


Upon boot as read-only startup proceeds up to where it's says;
--------------------

Mageia 3

[failed] systemd - Inserting module autofs4 // I don't care. I'm not using autofs.
[OK] systemd - sethostname to localhost
[OK] systemd - Initializing machine ID from random generator
[failed] systemd - Assertion 'close_nointr(fd) == 0' failed at src/shared/util.c:313 function close_nointr_nofail(). Aborting.
[failed] systemd - Caught <ABRT>. core dump failed
[failed] systemd - Freezing Execution.

--------------------------
The system is locked up at that point.

If I change the server's /etc/exports

Code: Select all

/diskless  10.0.0.0/24(ro,no_root_squash,no_subtree_check,async,insecure)

from ro to rw and let nfs mount the filesystem read/write, there is no problem and the system will boot and I'll get a login prompt.

Now the odd part, If I change the /etc/exports from rw back to ro and the re-export the filesystem from the fileserver with

Code: Select all

$ exportfs -a -r -v

Immediately systemd will generate this message over and over (usually in clumps of 4 every minute or so).

systemd-journald[XXXX] - Assertion 'close_nointr(fd) == 0' failed at src/shared/util.c:313, function close_nointr_nofail() Aborting.

If I reboot the diskless node now, it will proceed to boot properly to a login, but it will start generating the above messages again, this time it's not fatal. If I change the filesystem back to read/write (and reexport), the message goes away, put it back to readonly, and the message appears again.
I didn't see anything usual in the code for systemd in src/shared/util.c that looked unusual (although the code comments indicate some previous bug in handling the close_nointr.).