Mastering Collectl and Colplot for Linux System Performance Analysis

When troubleshooting complex Linux performance bottlenecks—such as sudden IO spikes, CPU throttling, or memory saturation—standard utilities like top, vmstat, and iostat are often insufficient. They require running multiple separate commands and do not easily correlate historical metrics across exact time slices.

Collectl is an incredibly lightweight, all-in-one performance monitoring utility that captures CPU, disk, memory, network, and NFS metrics simultaneously. When paired with Colplot, a web-based plotting engine, you can generate high-precision graphical analysis from your historical collectl raw logs.

In this guide, I will walk you through installing Colplot on Apache, automating raw log extraction, filtering specific storage subsystems (like EMC PowerPath and SCSI devices), and executing advanced forensic CLI queries.

Prerequisites

You will need a Linux server (CentOS/RHEL 7) with root access, configured with Apache HTTP Server and Gnuplot.

---

Step 1: Install Dependencies and Configure Colplot Web Server

Colplot uses Gnuplot under the hood to render performance graphs via a CGI script served by Apache.

Install Gnuplot and required utilities:

yum update -y
yum install gnuplot httpd -y

Configure Virtual Host

Create the Colplot directory structure and configure a dedicated Apache virtual host listening on port 8080.

homedir="/var/www/html/colplot"
logdir="/var/log/httpd/colplot"
domain="induslevel.com"
 
mkdir -p "${logdir}"
mkdir -p "${homedir}"
chown -R apache:apache "${logdir}"
chown -R apache:apache "${homedir}"
 
cat << EOF > /etc/httpd/conf.d/colplot-httpd.conf
Listen 8080
<VirtualHost *:8080>
  DocumentRoot /var/www/html/colplot
  ServerName colplot.${domain}
  CustomLog ${logdir}/access_log combined
  ErrorLog ${logdir}/error_log
  <Directory "${homedir}/">
    AllowOverride All
    Options ExecCGI
    AddHandler cgi-script .cgi
    DirectoryIndex index.cgi
    Options +FollowSymLinks +ExecCGI
    Require all granted
  </Directory>
</VirtualHost>
EOF

Start and enable Apache:

systemctl enable httpd
systemctl restart httpd

---

Step 2: Configure Colplot Plotting Directory and Automated Extraction

Colplot reads pre-processed plot files to render web graphs. We update the configuration to point to a dedicated plot directory and automate the extraction of raw compressed collectl logs.

plotdir="/var/log/collectl/plotfiles"
mkdir -p "$plotdir"

# Update colplot configuration
sed -i 's@^PlotDir.*=.*/var/log/collectl@PlotDir = /var/log/collectl/plotfiles@g' /etc/colplot.conf

Automated Log Plotting Script

To convert historical .raw.gz logs into plot files automatically in the background, execute this one-liner:

find /var/log/collectl -maxdepth 1 -mindepth 1 -type f -name "*\.raw\.gz" | sort -t'-' -k2 -n | while read line ; do
    collectl -p "${line}" -P -f /var/log/collectl/plotfiles -ocz 2>/dev/null & disown
done

---

Step 3: Advanced Forensic CLI Query Cheatsheet

While Colplot provides excellent visual graphs, the true power of Collectl lies in its highly flexible CLI filtering. Below is a curated cheatsheet of advanced query syntax for forensic metric analysis.

1. Time-Sliced CPU and Disk Extraction

Extract CPU (c), Network (n), and Disk (d) metrics across an exact time window:

collectl -scnd -oT --from 22:00 --thru 22:12 -p prod-db-server01-20210429-000000.raw.gz

2. Forensic Vmstat and Top Process Filtering

Replay historical logs imitating vmstat and process top behavior, filtering specifically for processes starting with f:

# Historical vmstat replay
collectl -oD -p test1-prod-db-server02-20210429-000000.raw.gz --from 22:00:00 --thru 22:14:00 --vmstat

# Historical top replay with process filtering
collectl -oD -p test1-prod-db-server02-20210429-000000.raw.gz --from 22:00:00 --thru 22:14:00 --top --procfilt f

# Historical top replay with full arguments
collectl -oD -p test1-prod-db-server02-20210429-000000.raw.gz --from 22:00:00 --thru 22:14:00 --top --full

3. Storage Subsystem Filtering (EMC PowerPath and SCSI)

Filter disk statistics specifically for EMC PowerPath enterprise SAN volumes or standard SCSI devices:

# Filter only EMC PowerPath devices
collectl -oD -p test1-prod-db-server02-20210429-000000.raw.gz --from 22:00:00 --thru 22:14:00 -sD --dskfilt emc

# Filter EMC PowerPath devices with extended IO options (z = skip zero activity)
collectl -oD -p test1-prod-db-server02-20210429-000000.raw.gz --from 22:00:00 --thru 22:14:00 -sD --dskfilt emc --dskopts z

# Filter standard SCSI disks (sdb)
collectl -scD -p prod-app-server01-20220410-000000.raw.gz -oT --dskfilt sdb --from 09:00:00 --thru 10:00:00

4. Multi-Subsystem Correlation

Correlate CPU, Disk, and Memory statistics simultaneously while excluding specific storage arrays:

# Exclude EMC and SAN storage arrays from CPU/Disk correlation
collectl -oD -p test1-prod-db-server02-20210429-000000.raw.gz --from 21:50:00 --thru 22:20:00 -scZ | grep cp | grep -v san | grep -v emc

# Analyze disk IO sizes across multiple device mappers
collectl -oD -p test1-prod-db-server02-20210429-000000.raw.gz --from 21:50:00 --thru 22:15:00 -sD --iosize --dskfilt ^emcpowerv,sd,dm --dskopts fiz

---

Step 4: Collectl Equivalent Command for iostat

A frequent requirement when analyzing storage latency is identifying specific disks experiencing high service times or saturation, similar to iostat -xz 1.

To achieve this with Collectl, filtering for EMC disks with extended IO size statistics and printing only occurrences where wait times exceed 100ms, execute this one-liner:

collectl -sD --dskfilt emc --dskopts o --iosize | awk '$2 > 100 {print;}'

By mastering these filtering flags, you can isolate performance anomalies down to the exact millisecond across any subsystem in your Linux infrastructure!

References

• Collectl & Colplot System Performance Analysis Tools