Major Changes and Features

Documentation about major changes and features is also included in the doc/features/ directory of GlusterFS repository.

Volume Snapshot

Volume snapshot provides a point-in-time copy of a GlusterFS volume. The snapshot is an online operation and hence filesystem data continues to be available for the clients while the snapshot is being taken.

For more information refer here.

User Serviceable Snapshots

User Serviceable Snapshots provides the ability for users to access snapshots of GlusterFS volumes without administrative intervention.

For more information refer here.

Erasure Coding

The new disperse translator provides the ability to perform erasure coding across nodes.

For more information refer here.

Granular locking support for management operations

Glusterd now holds a volume lock to support parallel management operations on different volumes.

Journaling enhancements (changelog xlator)

Introduction of history API to consume journal records which were persisted by the changelog translator. With this API, it's not longer required to perform an expensive filesystem crawl to identify changes. Geo-replication makes use of this (on [re]start) thereby optimizing remote replication for purges, hardlinks, etc.

Better Support for bricks with heterogeneous sizes

Prior to 3.6, bricks with heterogeneous sizes were treated as equal regardless of size, and would have been assigned an equal share of files. From 3.6, assignment of files to bricks will take into account the sizes of the bricks.

Improved SSL support

GlusterFS 3.6 provides better support to enable SSL on both management and data connections. This feature is currently being consumed by the GlusterFS native driver in OpenStack Manila.

Better peer identification

GlusterFS 3.6 improves peer identification. GlusterD will no longer complain when a mixture of FQDNs, shortnames and IP addresses are used. Changes done for this improvement have also laid down a base for improving multi network support in GlusterFS.

Meta translator

Meta translator provides a virtual interface for viewing internal state of translators.

Improved synchronous replication support (AFRv2)

The replication translator (AFR) in GlusterFS 3.6 has undergone a complete rewrite ( and is referred to as AFRv2.

From a user point of view, there is no change in the replication behaviour but there are some caveats to be noted from an admin point of view:

  • Lookups do not trigger meta-data and data self-heals anymore. They only trigger entry-self-heals. Data and meta-data are healed by the self-heal daemon only.

  • Bricks in a replica set do not mark any pending change log extended attributes for itself during pre or post op. They only mark it for other bricks in the replica set.

For e.g.: In a replica 2 volume, trusted.afr.<volname>-client-0 for brick-0 and trusted.afr.<volname>-client-1 for brick-1 will always be 0x000000000000000000000000.

  • If the post-op changelog updation does not complete successfully on a brick, a trusted.afr.dirty extended attribute is set on that brick.

Barrier translator

The barrier translator allows file operations to be temporarily 'paused' on GlusterFS bricks, which is needed for performing consistent snapshots of a GlusterFS volume.

For more information, see here.

Remove brick moves data by default

Prior to 3.6, volume remove-brick <volname> CLI would remove the brick from the volume without performing any data migration. Now the default behavior has been changed to perform data migration when this command is issued. Removing a brick without data migration can now be performed through volume remove-brick <volname> force interface.

Experimental Features

The following features are experimental with this release:

  • support for rdma volumes.
  • support for NUFA translator.
  • disk-encryption
  • On-Wire Compression + Decompression [CDC]

Porting Status

  • NetBSD and FreeBSD support is experimental, but regressions tests suggest that it is close to be fully supported. Please make sure you use latest NetBSD code from -current or netbsd-7 branches.

  • OSX support is in an alpha state. More testing will help in maturing this support.

Minor Improvements:

  • Introduction of server.anonuid and server.anongid options for root squashing

  • Root squashing doesn't happen for clients in trusted storage pool

  • Memory accounting of glusterfs processes has been enabled by default

  • The Gluster/NFS server now has support for setting access permissions on volumes with wildcard IP-addresses and IP-address/subnet (CIDR notation). More details and examples are in the commit message.

  • More preparation for better integration with the nfs-ganesha user-space NFS-server. The changes are mostly related to the handle-based functions in

  • A new logging framework that can suppress repetitive log messages and provide a dictionary of messages has been added. Few translators have now been integrated with the framework. More translators are expected to integrate with this framework in upcoming minor & major releases.

Known Issues:

  • The following configuration changes are necessary for qemu and samba integration with libgfapi to work seamlessly:

    1. gluster volume set <volname> server.allow-insecure on

    2. Edit /etc/glusterfs/glusterd.vol to contain this line: option rpc-auth-allow-insecure on

    Post 1, restarting the volume would be necessary: # gluster volume stop <volname> # gluster volume start <volname>

    Post 2, restarting glusterd would be necessary: # service glusterd restart

  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.

  • Renames happening on a file that is being migrated during rebalance will fail.

  • Dispersed volumes do not work with self-heal daemon. Self-healing is only activated when a damaged file or directory is accessed. To force a full self-heal or to replace a brick requires to traverse the file system from a mount point. This is the recommended command to do so:

    find <mount> -d -exec getfattr -h -n test {} \;
  • Quota on dispersed volumes is not correctly computed, allowing to store more data than specified. A workaround to this problem is to define a smaller quota based on this formula:

    Q' = Q / (N - R)

    Where Q is the desired quota value, Q' is the new quota value to use, N is the number of bricks per disperse set, and R is the redundancy.

Upgrading to 3.6.X

Before upgrading to 3.6 version of gluster from 3.4.X or 3.5.x, please take a look at following link: Upgrade Gluster to 3.6