Advanced Linux Network Forensics: Packet Captures and State Tracking

Troubleshooting sporadic network connectivity issues, dropped packets, or sudden firewall state table exhaustion across dozens of enterprise servers is a highly demanding sysadmin task. When diagnosing complex microservice communication or VoIP SIP signaling anomalies, running manual packet captures on individual machines is insufficient.

In this guide, I will explore an advanced network forensics workflow: automating circular tcpdump packet captures across server fleets using Ansible, alongside deep inspection of Netfilter Conntrack state tables to isolate dropped UDP connections and verify firewall connection tracking.

Prerequisites

You will need an administrative control node with Ansible installed, alongside root access to target Linux servers configured with tcpdump and conntrack-tools.

---

Step 1: Automating Fleet Packet Captures with Ansible

To diagnose intermittent connectivity issues across a distributed server fleet, we maintain a dedicated inventory file (tcpdumphost.inventory) and use Ansible ad-hoc commands to verify package availability and initiate long-running circular packet captures.

1. Verifying tcpdump Availability

Verify that tcpdump is installed across all target inventory nodes:

# Verify package availability via raw module
ansible -i /home/wazeem/tcpdumphost.inventory all --ssh-common-args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" -m raw -a "rpm -qa | egrep 'tcpdump'"

# Ensure package is installed via yum module
ansible -i /home/wazeem/tcpdumphost.inventory all --ssh-common-args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --become -m yum -a "name=tcpdump state=present"

---

Step 2: Circular Packet Capture Cheatsheet (tcpdump)

When capturing high-throughput network traffic over extended periods, writing to a single file will quickly exhaust local disk space. We configure tcpdump to execute circular ring-buffer captures, rotating files automatically based on size or time elapsed.

1. Size-Based Ring Buffer (Rotating every 100MB, Max 41 Files)

Capture HTTPS traffic on ens192, rotating files every 100MB and maintaining a maximum buffer of 41 files:

tcpdump -i ens192 -w /home/wazeem/networkissue.pcap -W 41 -G 1800 -C 100 -K -n port 443 and host 3ds2.sampledomain.com

2. Background Daemon Execution (nohup)

To execute long-running packet captures asynchronously across specific application ports (8092, 8480) without maintaining active SSH sessions, execute:

nohup tcpdump -i ens192 -C 1024 -W 2 -w /home/wazeem/appcapture.pcap port 8092 &
nohup tcpdump -i ens192 -C 1024 -W 2 -w /home/wazeem/appcapture.pcap port 8480 &

---

Step 3: Parsing Netfilter Conntrack State Tables

While tcpdump inspects raw wire packets, the Netfilter conntrack utility allows you to inspect the kernel's active connection tracking state table. This is critical for identifying NAT routing failures, unreplied connections, and firewall state table exhaustion.

1. Inspecting Real-Time Connection State Events

Monitor real-time connection creation (NEW), updates (UPDATE), and closures (DESTROY):

conntrack -E

[1609340393.532157]     [NEW] 1 30 src=10.10.240.183 dst=198.51.100.22 id=31510 [UNREPLIED] src=198.51.100.22 dst=10.10.240.183 type=0 code=0 id=31510
[1609340393.877052]  [UPDATE] 1 29 src=10.10.240.183 dst=198.51.100.22 type=8 code=0 id=31510 src=198.51.100.22 dst=10.10.240.183 type=0 code=0 id=31510
[1609340423.549566] [DESTROY] 1    src=10.10.240.183 dst=198.51.100.22 type=8 code=0 id=31510 src=198.51.100.22 dst=10.10.240.183 type=0 code=0 id=31510

2. Querying Active Protocol State Entries

Count total active UDP state entries versus fully established (ASSURED) connections:

# Count total active UDP connections
conntrack -L -p udp -n | wc -l

# Count established ASSURED UDP connections
conntrack -L -p udp -n -u ASSURED | wc -l

---

Step 4: Automated Forensic State Tracking Script

Below is an advanced Bash script used to isolate problematic UDP SIP connections (port 5060). It extracts all active connection attempts, compares them against fully established ASSURED connections, and prints an exact array of dropped or unreplied client IPs.

#!/bin/bash
# /usr/local/bin/conntrack-audit.sh
# Forensic script to isolate unreplied or dropped UDP SIP connections

CONTRACTALL="/tmp/conntrackallcon.log"
CONTRACTWORK="/tmp/conntrackworkcon.log"

# 1. Extract all active SIP connection attempts
conntrack -L -p udp -n | grep dport=5060 > $CONTRACTALL
allconnection=($(cat $CONTRACTALL | grep -v conntrack | awk '{print $5}' | awk -F= '{print $2}'))

# 2. Extract fully established ASSURED SIP connections
conntrack -L -p udp -n | grep dport=5060 | grep ASSURED > $CONTRACTWORK
workconnection=($(cat $CONTRACTWORK | grep -v conntrack | awk '{print $5}' | awk -F= '{print $2}'))

# 3. Isolate dropped or unreplied connections
dropped_connections=()
for i in "${allconnection[@]}"; do
    skip=
    for j in "${workconnection[@]}"; do
        [[ $i == $j ]] && { skip=1; break; }
    done
    [[ -n $skip ]] || dropped_connections+=("$i")
done

# 4. Output dropped connection IPs
echo "Problematic Unreplied SIP Client IPs:"
declare -p dropped_connections

References

• Netfilter Conntrack-Tools Documentation
• Ansible Ad-Hoc Commands Reference