Mageia forum

share the magic

Skip to content

GlusterFS Support?

This forum is dedicated to basic help and support :

Ask here your questions about basic installation and usage of Mageia. For example you may post here all your questions about getting Mageia isos and installing it, configuring your printer, using your word processor etc.

Try to ask your questions in the right sub-forum with as much details as you can gather. the more precise the question will be, the more likely you are to get a useful answer
Post a reply

GlusterFS Support?

Postby epretorious » May 5th, '12, 02:20

I've built a Gluster-based storage cluster on a pair of CentOS VM's (node1.example.com & node2.example.com)...

Code: Select all
[eric@node1 ~]$ sudo /usr/local/sbin/gluster --version
glusterfs 3.2.6 built on May  3 2012 15:53:02
...

[eric@node1 ~]$ sudo /usr/local/sbin/gluster peer status
Number of Peers: 1

Hostname: node2
Uuid: 793a855b-f4da-44f0-9de4-0be5c5e247a4
State: Peer in Cluster (Connected)

[eric@node1 ~]$ sudo /usr/local/sbin/gluster volume info mirror-1
Volume Name: mirror-1
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: node1:/srv
Brick2: node2:/srv
Options Reconfigured:
auth.allow: *

...and sometimes I'm able to mount the volume using NFS but not always:

  • Sometimes the connection times-out...

    Code: Select all
    [eric@localhost ~]$ sudo /usr/sbin/showmount -e node1.example.com
    Export list for node1.example.com:
    /mirror-1 *

    [eric@localhost ~]$ sudo mount -t nfs node1.example.com:/mirror-1 /mnt
    mount.nfs: Connection timed out

  • and other times the connection is refused:

    Code: Select all
    [eric@localhost ~]$ sudo mount -t nfs -o vers=3 node1.example.com:/mirror-1 /mnt
    mount.nfs: access denied by server while mounting node1.example.com:/mirror-1
And the Gluster client included in glusterfs-client-3.0.0-2.mga1 can't/won't mount the volume directly...

Code: Select all
[eric@localhost ~]$ locate mount.glusterfs
/sbin/mount.glusterfs
/usr/share/man/man8/mount.glusterfs.8.xz

[eric@localhost ~]$ rpm -qf /sbin/mount.glusterfs
glusterfs-client-3.0.0-2.mga1

[eric@localhost ~]$ date
Fri May  4 16:33:29 PDT 2012

[eric@localhost ~]$ sudo mount.glusterfs node1.example.com:/mirror-1 /mnt
error while getting volume file from server node1.example.com
Try `/usr/sbin/glusterfs --help' or `/usr/sbin/glusterfs --usage' for more
information.

[eric@localhost ~]$ sudo tail /var/log/glusterfs/mnt.log
...
[2012-05-04 16:33:23] W [socket.c:1407:socket_init] trans: disabling non-blocking IO
[2012-05-04 16:33:23] W [socket.c:1407:socket_init] trans: disabling non-blocking IO
[2012-05-04 16:33:23] E [socket.c:1088:socket_connect] trans: connection attempt failed (Connection refused)
[2012-05-04 16:33:23] E [glusterfsd.c:386:_get_specfp] glusterfs: error while getting volume file from server node1.example.com
[2012-05-04 16:33:23] E [glusterfsd.c:1216:main] glusterfs: exiting

[eric@localhost ~]$ sudo mount.glusterfs node1.example.com:/mirror-1 /mnt
error while getting volume file from server node1.example.com
Try `/usr/sbin/glusterfs --help' or `/usr/sbin/glusterfs --usage' for more
information.

[eric@localhost ~]$ rpm -qa | grep glusterfs
glusterfs-common-3.0.0-2.mga1
glusterfs-client-3.0.0-2.mga1
glusterfs-server-3.0.0-2.mga1
libglusterfs0-3.0.0-2.mga1

I've already verified that rpcbind/portmap is running on all three systems (i.e., the two storage nodes and the client)...

Code: Select all
[eric@node1 ~]$ sudo /sbin/service portmap status
portmap (pid 1739) is running...

[eric@node2 ~]$ sudo /sbin/service portmap status
portmap (pid 1741) is running...

[eric@localhost ~]$ sudo service rpcbind status
rpcbind (pid 2251) is running...

I've already verified that the firewall rules on all three systems (i.e., the two storage nodes and the client) have been flushed...

Code: Select all
[eric@node1 ~]$ sudo /sbin/service iptables status
Firewall is stopped.

[eric@node2 ~]$ sudo /sbin/service iptables status
Firewall is stopped.

[eric@localhost ~]$ sudo /sbin/service iptables status

Table: raw
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Table: nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Table: mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Table: filter
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I'm not sure what to make of the entries in the storage cluster's log files...

Code: Select all
[eric@node1 ~]$ date
Fri May  4 17:10:57 PDT 2012

[eric@node1 ~]$ sudo tail /usr/local/var/log/glusterfs/{mnt,nfs}.log
==> /usr/local/var/log/glusterfs/mnt.log <==
[2012-05-03 21:52:15.253073] I [glusterfsd.c:1493:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.2.6
[2012-05-03 21:52:15.283475] E [mount.c:532:fuse_mount_sys] 0-glusterfs-fuse: cannot open /dev/fuse (No such file or directory)
[2012-05-03 21:52:15.283710] I [mount.c:639:gf_fuse_mount] 0-glusterfs-fuse: direct mount failed (No such file or directory), retry to mount via fusermount
[2012-05-03 21:52:15.286034] E [mount.c:343:fuse_mount_fusermount] 0-glusterfs-fuse: failed to exec fusermount: No such file or directory
[2012-05-03 21:52:15.289658] E [mount.c:343:fuse_mount_fusermount] 0-glusterfs-fuse: failed to exec fusermount: No such file or directory
[2012-05-03 21:52:15.292071] E [mount.c:669:gf_fuse_mount] 0-glusterfs-fuse: mount failed
[2012-05-03 21:52:15.292569] E [xlator.c:1447:xlator_init] 0-fuse: Initialization of volume 'fuse' failed, review your volfile again

==> /usr/local/var/log/glusterfs/nfs.log <==
[2012-05-04 16:39:05.384565] I [afr-common.c:3141:afr_notify] 0-mirror-1-replicate-0: Subvolume 'mirror-1-client-1' came back up; going online.
[2012-05-04 16:39:05.387251] I [client-handshake.c:913:client_setvolume_cbk] 0-mirror-1-client-0: Connected to 10.0.0.1:24009, attached to remote volume '/srv'.
[2012-05-04 16:39:05.394769] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-mirror-1-replicate-0: added root inode
[2012-05-04 16:39:07.469623] W [socket.c:1494:__socket_proto_state_machine] 0-mirror-1-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.0.2:24009)
[2012-05-04 16:39:07.470303] I [client.c:1883:client_rpc_notify] 0-mirror-1-client-1: disconnected
[2012-05-04 16:39:11.404075] E [socket.c:1685:socket_connect_finish] 0-mirror-1-client-1: connection to 10.0.0.2:24009 failed (Connection refused)
[2012-05-04 16:39:14.397626] I [client-handshake.c:1090:select_server_supported_programs] 0-mirror-1-client-1: Using Program GlusterFS 3.2.6, Num (1298437), Version (310)
[2012-05-04 16:39:14.419861] I [client-handshake.c:913:client_setvolume_cbk] 0-mirror-1-client-1: Connected to 10.0.0.2:24009, attached to remote volume '/srv'.
[2012-05-04 16:39:35.466461] E [rpcsvc.c:500:nfs_rpcsvc_conn_peer_check_name] 0-nfsrpc: Failed to get remote addr: Temporary failure in name resolution
[2012-05-04 16:39:45.522037] E [rpcsvc.c:500:nfs_rpcsvc_conn_peer_check_name] 0-nfsrpc: Failed to get remote addr: Temporary failure in name resolution

And the client's log file isn't very helpful either...

Code: Select all
[eric@localhost ~]$ sudo tail /var/log/glusterfs/mnt.log
...
[2012-05-04 16:33:23] W [socket.c:1407:socket_init] trans: disabling non-blocking IO
[2012-05-04 16:33:23] W [socket.c:1407:socket_init] trans: disabling non-blocking IO
[2012-05-04 16:33:23] E [socket.c:1088:socket_connect] trans: connection attempt failed (Connection refused)
[2012-05-04 16:33:23] E [glusterfsd.c:386:_get_specfp] glusterfs: error while getting volume file from server node1.example.com
[2012-05-04 16:33:23] E [glusterfsd.c:1216:main] glusterfs: exiting

Ideas? Suggestions?
Last edited by epretorious on May 5th, '12, 03:12, edited 1 time in total.
epretorious
 
Posts: 44
Joined: Jan 6th, '12, 00:44
Top

Re: GlusterFS Support?

Postby epretorious » May 5th, '12, 02:55

epretorious wrote:...and sometimes I'm able to mount the volume using NFS but not always:

  • Sometimes the connection times-out...
  • and other times the connection is refused:
I'm not sure what to make of the entries in the storage cluster's log files...

Code: Select all
[eric@node1 ~]$ date

Fri May  4 17:10:57 PDT 2012

[eric@node1 ~]$ sudo tail /usr/local/var/log/glusterfs/{mnt,nfs}.log
...
==> /usr/local/var/log/glusterfs/nfs.log <==
...
[2012-05-04 16:39:35.466461] E [rpcsvc.c:500:nfs_rpcsvc_conn_peer_check_name] 0-nfsrpc: Failed to get remote addr: Temporary failure in name resolution
[2012-05-04 16:39:45.522037] E [rpcsvc.c:500:nfs_rpcsvc_conn_peer_check_name] 0-nfsrpc: Failed to get remote addr: Temporary failure in name resolution

  1. The LAN segment's gateway had failed (temporarily).
  2. This was preventing the nodes from reaching the nameserver (i.e., "Temporary failure in name resolution").
  3. This, in turn, was preventing the client from using NFS to mount the volume.
The Gluster client, however, still will not mount the remote volume:

Code: Select all
[eric@localhost ~]$ sudo mount.glusterfs node1.example.com:/mirror-1 /mnt
error while getting volume file from server node1.example.com
Try `/usr/sbin/glusterfs --help' or `/usr/sbin/glusterfs --usage' for more
information.

[eric@localhost ~]$ date
Fri May  4 17:52:31 PDT 2012

[eric@localhost ~]$ sudo tail /var/log/glusterfs/mnt.log
...
[2012-05-04 17:51:11] W [socket.c:1407:socket_init] trans: disabling non-blocking IO
[2012-05-04 17:51:11] W [socket.c:1407:socket_init] trans: disabling non-blocking IO
[2012-05-04 17:51:11] E [socket.c:1088:socket_connect] trans: connection attempt failed (Connection refused)
[2012-05-04 17:51:11] E [glusterfsd.c:386:_get_specfp] glusterfs: error while getting volume file from server node1.example.com
[2012-05-04 17:51:11] E [glusterfsd.c:1216:main] glusterfs: exiting
Last edited by epretorious on May 5th, '12, 03:15, edited 1 time in total.
epretorious
 
Posts: 44
Joined: Jan 6th, '12, 00:44
Top

Re: GlusterFS Support?

Postby epretorious » May 5th, '12, 03:12

FWIW: I'm able to mount the volume on the node using the latest version of the client...

Code: Select all
[eric@node1 ~]$ sudo /sbin/lsmod | grep fuse

[eric@node1 ~]$ sudo /sbin/modprobe fuse

[eric@node1 ~]$ sudo /sbin/lsmod | grep fuse
fuse                   49237  0

[eric@node1 ~]$ sudo mount -t glusterfs node1:/mirror-1 /mnt

[eric@node1 ~]$ sudo grep gluster /etc/mtab
glusterfs#node1:/mirror-1 /mnt fuse rw,allow_other,default_permissions,max_read=131072 0 0

...but still not able to mount the volume using Mageia's client...

Code: Select all
[eric@localhost ~]$ sudo /sbin/mount.glusterfs --version
glusterfs 3.0.0 built on Apr 10 2011 19:12:54
Repository revision: 2.0.1-886-g8379edd
...
epretorious
 
Posts: 44
Joined: Jan 6th, '12, 00:44
Top

Re: GlusterFS Support?

Postby epretorious » May 5th, '12, 03:29

A different approach...

Code: Select all
[eric@localhost ~]$ sudo glusterfs --debug -f /tmp/glusterfs.vol /mnt
[2012-05-04 19:09:09] D [glusterfsd.c:424:_get_specfp] glusterfs: loading volume file /tmp/glusterfs.vol
================================================================================
Version      : glusterfs 3.0.0 built on Apr 10 2011 19:12:54
git: 2.0.1-886-g8379edd
Starting Time: 2012-05-04 19:09:09
Command line : glusterfs --debug -f /tmp/glusterfs.vol /mnt
PID          : 30159
System name  : Linux
Nodename     : localhost.localdomain
Kernel Release : 2.6.38.8-desktop586-10.mga
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: volume mirror-1
  2:  type protocol/client
  3:  option transport-type tcp
  4:  option remote-host node1.example.com
  5:  option remote-subvolume mirror-1
  6: end-volume
+------------------------------------------------------------------------------+
[2012-05-04 19:09:09] D [glusterfsd.c:1335:main] glusterfs: running in pid 30159
[2012-05-04 19:09:09] D [client-protocol.c:6581:init] mirror-1: defaulting frame-timeout to 30mins
[2012-05-04 19:09:09] D [client-protocol.c:6592:init] mirror-1: defaulting ping-timeout to 42
[2012-05-04 19:09:09] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.0/transport/socket.so
[2012-05-04 19:09:09] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.0/transport/socket.so
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] N [glusterfsd.c:1361:main] glusterfs: Successfully started
[2012-05-04 19:09:09] E [socket.c:760:socket_connect_finish] mirror-1: connection to  failed (Connection refused)
[2012-05-04 19:09:09] D [fuse-bridge.c:3079:fuse_thread_proc] fuse:  pthread_cond_timedout returned non zero value ret: 0 errno: 0
[2012-05-04 19:09:09] N [fuse-bridge.c:2931:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16
[2012-05-04 19:09:09] E [socket.c:760:socket_connect_finish] mirror-1: connection to  failed (Connection refused)

[eric@localhost ~]$ ls -al /mnt
ls: cannot access /mnt: Transport endpoint is not connected

...with [effectively] the same result.
epretorious
 
Posts: 44
Joined: Jan 6th, '12, 00:44
Top

Re: GlusterFS Support?

Postby epretorious » May 5th, '12, 04:37

The Troubleshooting section of the Gluster User Guide recommends...
Connect to server failed SERVER-ADDRESS: Connection refused

GluserFS Server is not running or dead. Check your network connections and firewall settings. To check if the server is reachable, try:

Code: Select all
telnet IP-ADDRESS 6996

If the server is accessible, your `telnet' command should connect and block. If not you will see an error message such as telnet: Unable to connect to remote host: Connection refused. 6996 is the default GlusterFS port. If you have changed it, then use the corresponding port instead.

However, even when I use one of the nodes to mount the volume, Port #6996 does not appear in the list of open ports:

Code: Select all
[eric@node1 ~]$ sudo mount -t glusterfs node2:/mirror-1 /mnt

[eric@node1 ~]$ telnet 10.0.0.2 6996
Trying 10.0.0.2...
telnet: connect to address 10.0.0.2: Connection refused
telnet: Unable to connect to remote host: Connection refused

Code: Select all
[eric@node2 ~]$ sudo /bin/netstat -uanp | grep 6996
[eric@node2 ~]$
epretorious
 
Posts: 44
Joined: Jan 6th, '12, 00:44
Top
Post a reply

Return to Basic support

Who is online

Users browsing this forum: No registered users and 1 guest

Powered by phpBB® Forum Software © phpBB Group
cron