This section describes how to manage GlusterFS logs and most common troubleshooting scenarios related to GlusterFS.
- Managing GlusterFS Logs
- Troubleshooting POSIX ACLs
- Troubleshooting Hadoop Compatible Storage
- Troubleshooting File Locks
Managing GlusterFS Logs
Administrators can rotate the log file in a volume, as needed.
To rotate a log file
`# gluster volume log rotate `
For example, to rotate the log file on test-volume:
# gluster volume log rotate test-volume log rotate successful
Note When a log file is rotated, the contents of the current log file are moved to log-file- name.epoch-time-stamp.
Troubleshooting POSIX ACLs
This section describes the most common troubleshooting issues related to POSIX ACLs.
setfacl command fails with “setfacl: \<file or directory name\>: Operation not supported” error
You may face this error when the backend file systems in one of the servers is not mounted with the "-o acl" option. The same can be confirmed by viewing the following error message in the log file of the server "Posix access control list is not supported".
Solution: Remount the backend file system with "-o acl" option.
Troubleshooting Hadoop Compatible Storage
Problem: Running MapReduce job may throw exceptions if the time is out-of-sync on the hosts in the cluster.
Solution: Sync the time on all hosts using ntpd program.
Troubleshooting File Locks
In GlusterFS 3.3 you can use
statedump command to list the locks held
on files. The statedump output also provides information on each lock
with its range, basename, PID of the application holding the lock, and
so on. You can analyze the output to know about the locks whose
owner/application is no longer running or interested in that lock. After
ensuring that the no application is using the file, you can clear the
lock using the following
clear lock commands.
Perform statedump on the volume to view the files that are locked using the following command:
# gluster volume statedump inode
For example, to display statedump of test-volume:
# gluster volume statedump test-volume Volume statedump successful
The statedump files are created on the brick servers in the
/tmpdirectory or in the directory set using
server.statedump-pathvolume option. The naming convention of the dump file is
The following are the sample contents of the statedump file. It indicates that GlusterFS has entered into a state where there is an entry lock (entrylk) and an inode lock (inodelk). Ensure that those are stale locks and no resources own them.
[xlator.features.locks.vol-locks.inode] path=/ mandatory=0 entrylk-count=1 lock-dump.domain.domain=vol-replicate-0 xlator.feature.locks.lock-dump.domain.entrylk.entrylk(ACTIVE)=type=ENTRYLK_WRLCK on basename=file1, pid = 714782904, owner=ffffff2a3c7f0000, transport=0x20e0670, , granted at Mon Feb 27 16:01:01 2012 conn.2.bound_xl./gfs/brick1.hashsize=14057 conn.2.bound_xl./gfs/brick1.name=/gfs/brick1/inode conn.2.bound_xl./gfs/brick1.lru_limit=16384 conn.2.bound_xl./gfs/brick1.active_size=2 conn.2.bound_xl./gfs/brick1.lru_size=0 conn.2.bound_xl./gfs/brick1.purge_size=0 [conn.2.bound_xl./gfs/brick1.active.1] gfid=538a3d4a-01b0-4d03-9dc9-843cd8704d07 nlookup=1 ref=2 ia_type=1 [xlator.features.locks.vol-locks.inode] path=/file1 mandatory=0 inodelk-count=1 lock-dump.domain.domain=vol-replicate-0 inodelk.inodelk(ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 714787072, owner=00ffff2a3c7f0000, transport=0x20e0670, , granted at Mon Feb 27 16:01:01 2012
Clear the lock using the following command:
# gluster volume clear-locks
For example, to clear the entry lock on
# gluster volume clear-locks test-volume / kind granted entry file1 Volume clear-locks successful vol-locks: entry blocked locks=0 granted locks=1
Clear the inode lock using the following command:
# gluster volume clear-locks
For example, to clear the inode lock on
# gluster volume clear-locks test-volume /file1 kind granted inode 0,0-0 Volume clear-locks successful vol-locks: inode blocked locks=0 granted locks=1
You can perform statedump on test-volume again to verify that the above inode and entry locks are cleared.