How to Setup, Tune, and Troubleshoot a GlusterFS Distributed Storage Cluster on CentOS 7

GlusterFS is an open-source, distributed, scalable file system that aggregates disk storage resources from multiple servers into a single global namespace. Unlike clustered filesystems like GFS2 that require specialized Distributed Lock Managers (DLM) and Fibre Channel LUNs, GlusterFS operates purely over standard TCP/IP networks and local XFS filesystems (known as bricks).

In this guide, I will share the complete blueprint we used to deploy a 4-node GlusterFS cluster across two data centers (west-dc1 and east-dc1), establish granular firewalld rich rules, mount volumes via native FUSE, apply advanced caching and performance tuning, and troubleshoot active volume operations.

Prerequisites

You will need four CentOS 7 virtual machines with root access. Each VM requires a dedicated secondary drive (/dev/sdb) formatted with XFS.

Cluster Node Inventory

---

Step 1: Base System Preparation and Authentication

First, update the system packages, configure Grub timeouts, and disable SELinux on all four nodes.

# Generate SSH key pair
ssh-keygen -t rsa
echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0jlEIZdfm4D1LRKxQwfFz9cMLV8qxQJskqYKOspVT/oFF4p6XgnXeSAZdaWwcZh0wmNzU58PBIPp14j92xX0RGmyZvJBu/iHSiZqvEPxogOszeEQ5iOlgAPV1LivEiGuqOIoDzc3IP4AUsuGG6dnWh8e7HofdCOsL2r/ZX5pd0SY+IhIC3T+X6OKklWb+/mL1GCogY5waLkABYIQKYmYjaJ5E6jsYTtevRYT9Q+wK+2fujdCOYMw+luDKM7u3bQ2VC359ZabqB5082ke+xWu8HGmJ3GD5+FiF5qVSijl3dALjiabpnbOn7m/RzJ/1mi74IbGq4mUQE7ihAHC2GAK5Q== imported-openssh-key" >> /root/.ssh/authorized_keys

# Adjust Grub timeout
sed -i 's/^GRUB_TIMEOUT=.*/GRUB_TIMEOUT=2/g' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg

# Disable SELinux
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
reboot

Install Base Utilities and Join Domain

yum update -y
yum install -y rsync wget net-tools telnet tcpdump bash-completion ipa-client

# Enroll server in centralized LDAP directory
ipa-client-install --mkhomedir --enable-dns-updates

To ensure storage bricks resolve instantly during peer probing, add static entries to /etc/hosts:

cat >> /etc/hosts << "EOF"
10.110.10.120 west-dc1-gfs1.induslevel.com
10.110.10.121 west-dc1-gfs2.induslevel.com
10.112.10.120 east-dc1-gfs1.induslevel.com
10.112.10.121 east-dc1-gfs2.induslevel.com
EOF

---

Step 2: Storage Brick Formatting and Mounting

GlusterFS stores data inside underlying directories called bricks. We format /dev/sdb as XFS and mount it persistently.

Execute the following on all four nodes:

# Format partition as XFS
mkfs.xfs -f /dev/sdb

# Create mount directory for brick storage
mkdir -p /data

# Add persistent mount entry to /etc/fstab
echo "/dev/sdb /data xfs defaults 0 0" >> /etc/fstab
mount /data
df -h /data

---

Step 3: Install GlusterFS Packages

To maintain absolute version consistency across all nodes, we download and install the GlusterFS 5.0 RPM packages locally from the storage repository:

mkdir -p /root/gluster-packages && cd /root/gluster-packages

# Download required RPMs (or install directly via yum if repository is enabled)
yum install -y glusterfs-libs-5.0-1.el7.x86_64.rpm \
    glusterfs-cli-5.0-1.el7.x86_64.rpm \
    glusterfs-client-xlators-5.0-1.el7.x86_64.rpm \
    glusterfs-5.0-1.el7.x86_64.rpm \
    glusterfs-api-5.0-1.el7.x86_64.rpm \
    glusterfs-fuse-5.0-1.el7.x86_64.rpm \
    rpcbind \
    userspace-rcu-0.10.0-3.el7.x86_64.rpm \
    glusterfs-server-5.0-1.el7.x86_64.rpm

Enable and start the GlusterFS management daemon (glusterd):

systemctl enable glusterd.service
systemctl start glusterd.service
systemctl status glusterd.service

---

Step 4: Granular Firewalld Rich Rules Configuration

GlusterFS requires communication across several management and brick ports:

• 24007/tcp: GlusterFS Management Daemon (glusterd)
• 24008/tcp: GlusterFS RDMA port management
• 24009/tcp: GlusterFS Events Daemon (glustereventsd)
• 49152-49664/tcp: Brick access ports

To enforce strict network security, we use firewalld rich rules to restrict management and brick access exclusively between authorized cluster nodes.

Global Management and Brick Access (All Nodes)

firewall-cmd --zone=public --add-port=24007-24009/tcp --permanent
firewall-cmd --zone=public --add-port=49152-49153/tcp --permanent
firewall-cmd --reload && firewall-cmd --list-all

Peer-Specific Rich Rules (Node 1 - west-dc1-gfs1)

firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=10.110.10.121 port port=24007 protocol=tcp accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=10.110.10.121 port port=24008 protocol=tcp accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=10.110.10.121 port port=24009 protocol=tcp accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=10.110.10.121 port port=49152 protocol=tcp accept'
firewall-cmd --reload && firewall-cmd --list-all

(Repeat rich rule configuration on Nodes 2, 3, and 4 targeting their respective peer IP addresses).

---

Step 5: Cluster Probing and Volume Creation

With network communication unlocked, we probe peer nodes and establish our distributed replica volumes.

1. Create West Data Center 1 Cluster Volume (Node 1)

gluster peer probe west-dc1-gfs2.induslevel.com
gluster peer status

# Create replica volume across both West DC1 nodes
gluster volume create testvol replica 2 transport tcp west-dc1-gfs1.induslevel.com:/data west-dc1-gfs2.induslevel.com:/data force
gluster volume start testvol
gluster volume status testvol
gluster volume info
gluster volume set testvol auth.allow 10.110.10.120,10.110.10.121

2. Create East Data Center 1 Cluster Volume (Node 3)

gluster peer probe east-dc1-gfs2.induslevel.com
gluster peer status

# Create replica volume across both East DC1 nodes
gluster volume create testvol replica 2 transport tcp east-dc1-gfs1.induslevel.com:/data east-dc1-gfs2.induslevel.com:/data force
gluster volume start testvol
gluster volume status testvol
gluster volume info
gluster volume set testvol auth.allow 10.112.10.120,10.112.10.121

---

Step 6: Mounting Volumes via Native FUSE

Mounting GlusterFS volumes via native FUSE (Filesystem in Userspace) provides significantly better performance and failover capabilities than standard NFS.

Execute the following on all respective nodes:

# Create mount directory
mkdir -p /export/home/chroot

# Mount locally via FUSE
mount.glusterfs localhost:/testvol /export/home/chroot

Persistent /etc/fstab Configuration

Add the persistent FUSE mount entry to /etc/fstab on each server:

# On West DC1 Node 1
west-dc1-gfs1.induslevel.com:/testvol /export/home/chroot glusterfs defaults,_netdev 0 0

# On West DC1 Node 2
west-dc1-gfs2.induslevel.com:/testvol /export/home/chroot glusterfs defaults,_netdev 0 0

# On East DC1 Node 1
east-dc1-gfs1.induslevel.com:/testvol /export/home/chroot glusterfs defaults,_netdev 0 0

# On East DC1 Node 2
east-dc1-gfs2.induslevel.com:/testvol /export/home/chroot glusterfs defaults,_netdev 0 0

---

Step 7: Advanced GlusterFS Performance Tuning

Out of the box, GlusterFS prioritizes stability over raw throughput. Applying targeted performance tuning parameters drastically accelerates read and write speeds, especially for high-concurrency workloads like VDI or web applications.

Run the following tuning commands against your volume (testvol):

# Enable Readdir-Ahead and Parallel-Readdir for rapid directory listings
gluster volume set testvol performance.readdir-ahead on
gluster volume set testvol performance.parallel-readdir on

# Expand Cache Size from 32MB to 2GB
gluster volume set testvol performance.cache-size 2048MB

# Optimize Metadata Caching and Inode LRU Limits
gluster volume set testvol group metadata-cache
gluster volume set testvol network.inode-lru-limit 1000000

# Increase IO Threads and Write-Behind Window Size
gluster volume set testvol performance.io-cache on
gluster volume set testvol performance.io-thread-count 32
gluster volume set testvol performance.write-behind-window-size 10MB

# Disable unnecessary NFS daemons to conserve resources
gluster volume set testvol nfs.disable on

Reverting Tuning Parameters (Rollback)

If your application experiences cache invalidation conflicts, you can safely revert to default parameters:

gluster volume set testvol performance.cache-invalidation false
gluster volume set testvol performance.md-cache-timeout 1
gluster volume set testvol network.inode-lru-limit 16384
gluster volume set testvol performance.stat-prefetch on
gluster volume set testvol performance.cache-invalidation false
gluster volume set testvol features.cache-invalidation off
gluster volume set testvol features.cache-invalidation-timeout 60

---

Step 8: Real-Time Troubleshooting and Monitoring

When managing GlusterFS in production, understanding how to monitor internal volume operations is essential for diagnosing latency or bottlenecks.

Here is a cheatsheet of essential troubleshooting commands:

# 1. Inspect Real-Time Write Performance (Top 10 operations)
gluster volume top testvol write-perf list-cnt 10

# 2. Inspect Real-Time Read Performance (Top 20 operations)
gluster volume top testvol read-perf list-cnt 20

# 3. Check Open File Descriptors and Client Connections
gluster volume top testvol open list-cnt 10
gluster volume status testvol clients

# 4. Check Memory and Inode Consumption across Bricks
gluster volume status testvol mem
gluster volume status testvol inode
gluster volume status testvol fd

# 5. Review Complete Volume Topology and Detailed Status
gluster volume status all detail

Your GlusterFS distributed storage architecture is now fully deployed, secured via rich firewall rules, accelerated with advanced performance profiles, and fully monitorable!

References

• GlusterFS Performance Tuning Guide
• CentOS Storage SIG Gluster Quickstart