/usr appears to be mounted readonly.

[linuxdad]

I needed to reboot the server last night, as I was attempting to add a secondary route to the box, and it quit responding to me.

Now I am getting the following:

urpmi libiptables-devel
Error: /usr appears to be mounted read-only.


Is a reboot going to clear this up? Why is this occurring?

[isadora]

OK, I found the thread which explains that separate filesystems .... I don't agree (I am Running a Server for crying out loud!) - viewtopic.php?f=15&t=1644&p=11856&hilit=%2Fusr+mounted+read+only#p11856

before remounting the /usr in rw mode, I cannot even use:

perl Makefile.PL correctly:

Code: Select all


[root@ns IPTables-libiptc-0.52c]# perl Makefile.PL
Detect iptables version via command: "iptables -V"
Detect the xtables libdir path of: "iptables"
 - detected xtables libdir: "/usr/libexec/xtables"
Detected iptables version: 1.4.17 (vercode:66577)
Using iptables version: 1.4.17
Checking for header files
 - detected include path: "/usr/include"
 - check-header: xtables.h

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-        *** ERROR: Missing header file ***       -=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

   Cannot find required header file: "xtables.h"

   You must install the systems software package
   containing the header file, usually a "dev" package.

   Hint, under Debian/Ubuntu install "iptables-dev" package
     apt-get install iptables-dev
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Generating a Unix-style Makefile
Writing Makefile for IPTables::libiptc
Unable to open MakeMaker.tmp: Read-only file system at /root/perl5/lib/perl5/ExtUtils/MakeMaker.pm line 1139.

Please tell me that this is NOT a Feature. (it sure is an undesireable one anyway).

[doktor5000]

Sorry, but I don't get your issue at all. Please show the output of

Code: Select all

df /usr

and

Code: Select all

mount

to begin with, also the fstab entry for the partition that carries /usr.

Did you already try an fsck on that filesystem, or to remount it read-write?

[linuxdad]

Thank you Doctor.

First of all I am running a server. This is not a Desktop, and utilizing different partitions improves performance, and restrict logfiles from consuming more resources than they should. Per your request:

Code: Select all

[root@ns ~]# df
Filesystem                                    Size  Used Avail Use% Mounted on
rootfs                                        4.5G  329M  3.9G   8% /
devtmpfs                                      4.0G     0  4.0G   0% /dev
tmpfs                                         4.0G   84K  4.0G   1% /dev/shm
tmpfs                                         4.0G  700K  4.0G   1% /run
/dev/sda1                                     4.5G  329M  3.9G   8% /
/dev/sda6                                     7.6G  4.0G  3.3G  56% /usr
tmpfs                                         4.0G     0  4.0G   0% /sys/fs/cgroup
/dev/sda8                                     4.4G  9.4M  4.2G   1% /tmp
/dev/sda7                                      16G  2.2G   13G  15% /var
/dev/sda1                                     4.5G  329M  3.9G   8% /var/lib/named/etc/rndc.key
/dev/sda1                                     4.5G  329M  3.9G   8% /var/lib/named/etc/named.iscdlv.key
/dev/sda1                                     4.5G  329M  3.9G   8% /var/lib/named/etc/named.root.key
/dev/sda6                                     7.6G  4.0G  3.3G  56% /var/lib/named/usr/lib/bind
/dev/sda6                                     7.6G  4.0G  3.3G  56% /var/lib/named/usr/lib/openssl
/dev/sda7                                      16G  2.2G   13G  15% /var/lib/named/var/named
ns5.gigabit.somehost.com:/sysadm           32G  3.8G   27G  13% /sysadm
ns5.gigabit.somehost.com:/home             44G   22G   23G  50% /home
ns5.gigabit.somehost.com:/var/spool/mail   11G  2.0G  8.3G  20% /mail/spool/mail
[root@ns ~]# mount
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4147120k,nr_inodes=185975,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,data=ordered)
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,mode=755)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda8 on /tmp type ext4 (rw,relatime,data=ordered)
/dev/sda7 on /var type ext4 (rw,relatime,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sda1 on /var/lib/named/etc/rndc.key type ext4 (ro,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/named.iscdlv.key type ext4 (ro,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/named.root.key type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/bind type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/openssl type ext4 (ro,relatime,data=ordered)
/dev/sda7 on /var/lib/named/var/named type ext4 (rw,relatime,data=ordered)
ns5.gigabit.ABS-CompTech.com:/sysadm on /sysadm type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)
ns5.gigabit.ABS-CompTech.com:/home on /home type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)
ns5.gigabit.ABS-CompTech.com:/var/spool/mail on /mail/spool/mail type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)


I guess that mounting /usr is a partial security feature/enhancement/bug, but I also believe that it will be a hinderance for updates on the server daily as well.

How can I restart this beast, and have /usr in rw mode?

[jiml8]

Could be that the system remounted /usr as read only because it detected a filesystem problem, or there was an I/O problem. You should check for this before doing anything else.

Beyond that, you can always remount the filesystem as read-write

Code: Select all

mount -o remount,rw /usr


This will fail if you have a filesystem or hard drive error.

[doktor5000]

Server or not doesn't matter. And i'll not comment on those improvements in performance ...

/dev/sda6 7.6G 4.0G 3.3G 56% /usr
[...]
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)


I guess that mounting /usr is a partial security feature/enhancement/bug

Huh? Probably you shouldn't guess that much. According to the output you posted /usr is mounted readwrite. You also forgot the output of

Code: Select all

df /usr

Maybe something broke, you should check /etc/mtab and /proc/mounts if you see any differences. Apart from that I'd probably reboot into an emergency shell or single user mode and fsck that thing, like previously mentioned already.

[jiml8]

I would expect that checking /usr would not be possible when booted into that system, unless you chroot to a different /. Best to load a Linux Live CD for that test.

[linuxdad]

I am still experiencing this issue.

Investigating this further brings me the following information:

Code: Select all

[root@ns log]# mount | grep sda6
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/bind type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/openssl type ext4 (ro,relatime,data=ordered)

To me, it would appear that the Chrooted named directory is now associated with the /usr filesystem?

I am not exactly certain how this occurs, because in my mind, the chroot for named is /var/lib/named.

so how does the chroot jump from /var to /usr?

Here's some additional information on the matter.

Code: Select all

[root@ns log]# df
Filesystem                                    Size  Used Avail Use% Mounted on
rootfs                                        4.5G  518M  3.7G  13% /
devtmpfs                                      4.0G     0  4.0G   0% /dev
tmpfs                                         4.0G   84K  4.0G   1% /dev/shm
tmpfs                                         4.0G  716K  4.0G   1% /run
/dev/sda1                                     4.5G  518M  3.7G  13% /
/dev/sda6                                     7.6G  4.2G  3.1G  58% /usr
tmpfs                                         4.0G     0  4.0G   0% /sys/fs/cgroup
/dev/sda7                                      16G  2.5G   12G  18% /var
/dev/sda8                                     4.4G  9.4M  4.2G   1% /tmp
/dev/sda1                                     4.5G  518M  3.7G  13% /var/lib/named/etc/rndc.key
/dev/sda1                                     4.5G  518M  3.7G  13% /var/lib/named/etc/named.iscdlv.key
/dev/sda1                                     4.5G  518M  3.7G  13% /var/lib/named/etc/named.root.key
/dev/sda6                                     7.6G  4.2G  3.1G  58% /var/lib/named/usr/lib/bind
/dev/sda6                                     7.6G  4.2G  3.1G  58% /var/lib/named/usr/lib/openssl
/dev/sda7                                      16G  2.5G   12G  18% /var/lib/named/var/named
ns5.gigabit.ABS-CompTech.com:/sysadm           32G  3.8G   27G  13% /sysadm
ns5.gigabit.ABS-CompTech.com:/var/spool/mail   11G  616M  9.7G   6% /mail/spool/mail
ns5.gigabit.ABS-CompTech.com:/home             44G   22G   23G  50% /home

and the mount points:

[root@ns log]# mount
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4147120k,nr_inodes=185975,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,data=ordered)
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,mode=755)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda7 on /var type ext4 (rw,relatime,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sda8 on /tmp type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/rndc.key type ext4 (ro,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/named.iscdlv.key type ext4 (ro,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/named.root.key type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/bind type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/openssl type ext4 (ro,relatime,data=ordered)
/dev/sda7 on /var/lib/named/var/named type ext4 (rw,relatime,data=ordered)
ns5.gigabit.ABS-CompTech.com:/sysadm on /sysadm type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)
ns5.gigabit.ABS-CompTech.com:/var/spool/mail on /mail/spool/mail type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)
ns5.gigabit.ABS-CompTech.com:/home on /home type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)

Now here's another question, what is the cgroup collection?

Thank you.

[doktor5000]

Code: Select all

[root@ns log]# mount | grep sda6
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/bind type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/openssl type ext4 (ro,relatime,data=ordered)

To me, it would appear that the Chrooted named directory is now associated with the /usr filesystem?

I am not exactly certain how this occurs, because in my mind, the chroot for named is /var/lib/named.

so how does the chroot jump from /var to /usr?

By doing bind mounts, i.e. a filesystem or a part of an existing filesystem gets remounted elsewhere.
Check e.g. http://docs.1h.com/Bind_mounts or https://access.redhat.com/site/document ... order.html or https://access.redhat.com/site/document ... order.html

What could maybe happen, if on reboot/shutdown you or the system tries to unmount /usr itself before the bind mounts are unmounted, that will not be possible, even when forcing it (device is busy).

Now here's another question, what is the cgroup collection?

Thank you.

cgroups are a relatively recent linux thing, short for control groups, mainly used by systemd.
Check http://0pointer.de/blog/projects/cgroup ... roups.html or https://wiki.archlinux.org/index.php/cgroups