Page 1 of 1

/usr appears to be mounted readonly.

PostPosted: Dec 24th, '13, 16:09
by linuxdad
I needed to reboot the server last night, as I was attempting to add a secondary route to the box, and it quit responding to me.

Now I am getting the following:

urpmi libiptables-devel
Error: /usr appears to be mounted read-only.


Is a reboot going to clear this up? Why is this occurring?

Re: /usr appears to be mounted readonly.

PostPosted: Dec 24th, '13, 16:17
by linuxdad
OK, I found the thread which explains that separate filesystems .... I don't agree (I am Running a Server for crying out loud!) - viewtopic.php?f=15&t=1644&p=11856&hilit=%2Fusr+mounted+read+only#p11856

before remounting the /usr in rw mode, I cannot even use:

perl Makefile.PL correctly:
Code: Select all

[root@ns IPTables-libiptc-0.52c]# perl Makefile.PL
Detect iptables version via command: "iptables -V"
Detect the xtables libdir path of: "iptables"
 - detected xtables libdir: "/usr/libexec/xtables"
Detected iptables version: 1.4.17 (vercode:66577)
Using iptables version: 1.4.17
Checking for header files
 - detected include path: "/usr/include"
 - check-header: xtables.h

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-        *** ERROR: Missing header file ***       -=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

   Cannot find required header file: "xtables.h"

   You must install the systems software package
   containing the header file, usually a "dev" package.

   Hint, under Debian/Ubuntu install "iptables-dev" package
     apt-get install iptables-dev
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Generating a Unix-style Makefile
Writing Makefile for IPTables::libiptc
Unable to open MakeMaker.tmp: Read-only file system at /root/perl5/lib/perl5/ExtUtils/MakeMaker.pm line 1139.



Please tell me that this is NOT a Feature. (it sure is an undesireable one anyway).

Re: /usr appears to be mounted readonly.

PostPosted: Dec 24th, '13, 18:15
by doktor5000
Sorry, but I don't get your issue at all. Please show the output of
Code: Select all
df /usr
and
Code: Select all
mount
to begin with, also the fstab entry for the partition that carries /usr.

Did you already try an fsck on that filesystem, or to remount it read-write?

Re: /usr appears to be mounted readonly.

PostPosted: Dec 24th, '13, 18:22
by linuxdad
Thank you Doctor.

First of all I am running a server. This is not a Desktop, and utilizing different partitions improves performance, and restrict logfiles from consuming more resources than they should. Per your request:

Code: Select all
[root@ns ~]# df
Filesystem                                    Size  Used Avail Use% Mounted on
rootfs                                        4.5G  329M  3.9G   8% /
devtmpfs                                      4.0G     0  4.0G   0% /dev
tmpfs                                         4.0G   84K  4.0G   1% /dev/shm
tmpfs                                         4.0G  700K  4.0G   1% /run
/dev/sda1                                     4.5G  329M  3.9G   8% /
/dev/sda6                                     7.6G  4.0G  3.3G  56% /usr
tmpfs                                         4.0G     0  4.0G   0% /sys/fs/cgroup
/dev/sda8                                     4.4G  9.4M  4.2G   1% /tmp
/dev/sda7                                      16G  2.2G   13G  15% /var
/dev/sda1                                     4.5G  329M  3.9G   8% /var/lib/named/etc/rndc.key
/dev/sda1                                     4.5G  329M  3.9G   8% /var/lib/named/etc/named.iscdlv.key
/dev/sda1                                     4.5G  329M  3.9G   8% /var/lib/named/etc/named.root.key
/dev/sda6                                     7.6G  4.0G  3.3G  56% /var/lib/named/usr/lib/bind
/dev/sda6                                     7.6G  4.0G  3.3G  56% /var/lib/named/usr/lib/openssl
/dev/sda7                                      16G  2.2G   13G  15% /var/lib/named/var/named
ns5.gigabit.somehost.com:/sysadm           32G  3.8G   27G  13% /sysadm
ns5.gigabit.somehost.com:/home             44G   22G   23G  50% /home
ns5.gigabit.somehost.com:/var/spool/mail   11G  2.0G  8.3G  20% /mail/spool/mail
[root@ns ~]# mount
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4147120k,nr_inodes=185975,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,data=ordered)
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,mode=755)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda8 on /tmp type ext4 (rw,relatime,data=ordered)
/dev/sda7 on /var type ext4 (rw,relatime,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sda1 on /var/lib/named/etc/rndc.key type ext4 (ro,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/named.iscdlv.key type ext4 (ro,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/named.root.key type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/bind type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/openssl type ext4 (ro,relatime,data=ordered)
/dev/sda7 on /var/lib/named/var/named type ext4 (rw,relatime,data=ordered)
ns5.gigabit.ABS-CompTech.com:/sysadm on /sysadm type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)
ns5.gigabit.ABS-CompTech.com:/home on /home type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)
ns5.gigabit.ABS-CompTech.com:/var/spool/mail on /mail/spool/mail type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)


I guess that mounting /usr is a partial security feature/enhancement/bug, but I also believe that it will be a hinderance for updates on the server daily as well.

How can I restart this beast, and have /usr in rw mode?

Re: /usr appears to be mounted readonly.

PostPosted: Dec 24th, '13, 19:24
by jiml8
Could be that the system remounted /usr as read only because it detected a filesystem problem, or there was an I/O problem. You should check for this before doing anything else.

Beyond that, you can always remount the filesystem as read-write
Code: Select all
mount -o remount,rw /usr


This will fail if you have a filesystem or hard drive error.

Re: /usr appears to be mounted readonly.

PostPosted: Dec 24th, '13, 20:26
by doktor5000
Server or not doesn't matter. And i'll not comment on those improvements in performance ...
linuxdad wrote:
/dev/sda6 7.6G 4.0G 3.3G 56% /usr
[...]
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)


I guess that mounting /usr is a partial security feature/enhancement/bug

Huh? Probably you shouldn't guess that much. According to the output you posted /usr is mounted readwrite. You also forgot the output of
Code: Select all
df /usr


Maybe something broke, you should check /etc/mtab and /proc/mounts if you see any differences. Apart from that I'd probably reboot into an emergency shell or single user mode and fsck that thing, like previously mentioned already.

Re: /usr appears to be mounted readonly.

PostPosted: Dec 26th, '13, 15:52
by jiml8
I would expect that checking /usr would not be possible when booted into that system, unless you chroot to a different /. Best to load a Linux Live CD for that test.

Re: /usr appears to be mounted readonly.

PostPosted: Jan 9th, '14, 18:15
by linuxdad
I am still experiencing this issue.

Investigating this further brings me the following information:

Code: Select all
[root@ns log]# mount | grep sda6
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/bind type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/openssl type ext4 (ro,relatime,data=ordered)


To me, it would appear that the Chrooted named directory is now associated with the /usr filesystem?

I am not exactly certain how this occurs, because in my mind, the chroot for named is /var/lib/named.

so how does the chroot jump from /var to /usr?

Here's some additional information on the matter.

Code: Select all
[root@ns log]# df
Filesystem                                    Size  Used Avail Use% Mounted on
rootfs                                        4.5G  518M  3.7G  13% /
devtmpfs                                      4.0G     0  4.0G   0% /dev
tmpfs                                         4.0G   84K  4.0G   1% /dev/shm
tmpfs                                         4.0G  716K  4.0G   1% /run
/dev/sda1                                     4.5G  518M  3.7G  13% /
/dev/sda6                                     7.6G  4.2G  3.1G  58% /usr
tmpfs                                         4.0G     0  4.0G   0% /sys/fs/cgroup
/dev/sda7                                      16G  2.5G   12G  18% /var
/dev/sda8                                     4.4G  9.4M  4.2G   1% /tmp
/dev/sda1                                     4.5G  518M  3.7G  13% /var/lib/named/etc/rndc.key
/dev/sda1                                     4.5G  518M  3.7G  13% /var/lib/named/etc/named.iscdlv.key
/dev/sda1                                     4.5G  518M  3.7G  13% /var/lib/named/etc/named.root.key
/dev/sda6                                     7.6G  4.2G  3.1G  58% /var/lib/named/usr/lib/bind
/dev/sda6                                     7.6G  4.2G  3.1G  58% /var/lib/named/usr/lib/openssl
/dev/sda7                                      16G  2.5G   12G  18% /var/lib/named/var/named
ns5.gigabit.ABS-CompTech.com:/sysadm           32G  3.8G   27G  13% /sysadm
ns5.gigabit.ABS-CompTech.com:/var/spool/mail   11G  616M  9.7G   6% /mail/spool/mail
ns5.gigabit.ABS-CompTech.com:/home             44G   22G   23G  50% /home


and the mount points:

[root@ns log]# mount
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4147120k,nr_inodes=185975,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,data=ordered)
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,mode=755)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda7 on /var type ext4 (rw,relatime,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sda8 on /tmp type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/rndc.key type ext4 (ro,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/named.iscdlv.key type ext4 (ro,relatime,data=ordered)
/dev/sda1 on /var/lib/named/etc/named.root.key type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/bind type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/openssl type ext4 (ro,relatime,data=ordered)
/dev/sda7 on /var/lib/named/var/named type ext4 (rw,relatime,data=ordered)
ns5.gigabit.ABS-CompTech.com:/sysadm on /sysadm type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)
ns5.gigabit.ABS-CompTech.com:/var/spool/mail on /mail/spool/mail type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)
ns5.gigabit.ABS-CompTech.com:/home on /home type nfs (rw,nosuid,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.15,mountvers=3,mountport=57092,mountproto=udp,local_lock=none,addr=192.168.50.15)


Now here's another question, what is the cgroup collection?

Thank you.

Re: /usr appears to be mounted readonly.

PostPosted: Jan 9th, '14, 20:07
by doktor5000
linuxdad wrote:
Code: Select all
[root@ns log]# mount | grep sda6
/dev/sda6 on /usr type ext4 (rw,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/bind type ext4 (ro,relatime,data=ordered)
/dev/sda6 on /var/lib/named/usr/lib/openssl type ext4 (ro,relatime,data=ordered)


To me, it would appear that the Chrooted named directory is now associated with the /usr filesystem?

I am not exactly certain how this occurs, because in my mind, the chroot for named is /var/lib/named.

so how does the chroot jump from /var to /usr?


By doing bind mounts, i.e. a filesystem or a part of an existing filesystem gets remounted elsewhere.
Check e.g. http://docs.1h.com/Bind_mounts or https://access.redhat.com/site/document ... order.html or https://access.redhat.com/site/document ... order.html

What could maybe happen, if on reboot/shutdown you or the system tries to unmount /usr itself before the bind mounts are unmounted, that will not be possible, even when forcing it (device is busy).


linuxdad wrote:Now here's another question, what is the cgroup collection?

Thank you.

cgroups are a relatively recent linux thing, short for control groups, mainly used by systemd.
Check http://0pointer.de/blog/projects/cgroup ... roups.html or https://wiki.archlinux.org/index.php/cgroups