Building Resilient Reverse SSH Tunnels with AutoSSH and Ansible

When managing a highly distributed fleet of remote thin clients or employee workstations behind restricted NAT firewalls, establishing secure, persistent remote support sessions is incredibly difficult. Standard SSH connections drop when network interruptions occur, and inbound firewall rules prevent direct administration.

AutoSSH solves this by monitoring active SSH connections and automatically restarting them if they fail or drop. By pairing AutoSSH on remote workstations with a hardened, central reverse SSH tunneling server, you can dynamically expose remote administration ports and manage your entire fleet simultaneously using Ansible.

In this guide, I will share the complete architectural blueprint we used to deploy a dedicated reverse SSH tunneling server, establish AutoSSH systemd services on target workstations, discover dynamic local forwarding ports via automated shell scripts, and execute bulk administration playbooks.

Prerequisites

You will need a public-facing Linux server acting as the reverse tunneling gateway (remotesupport.induslevel.com), alongside remote thin clients configured with root privileges and network reachability over HTTPS port 443.

---

Step 1: Configuring the Hardened Reverse Tunneling Server

On your public gateway server, create an unprivileged system user without shell access. This account (tclient) is used exclusively by remote clients to establish reverse forwarding tunnels.

# Create unprivileged user without shell access
useradd -s /bin/false tclient

# Create SSH directory and configure permissions
mkdir -p /home/tclient/.ssh
chmod 700 /home/tclient/.ssh

Place your client public keys inside /home/tclient/.ssh/authorized_keys and secure ownership:

chown -R tclient:tclient /home/tclient/.ssh
chmod 600 /home/tclient/.ssh/authorized_keys

Structuring the Hardened SSH Daemon Configuration

To isolate tunneling traffic from standard administrative SSH access, configure a dedicated OpenSSH daemon configuration file (/etc/ssh/sshd_config_support) listening on port 443:

# /etc/ssh/sshd_config_support
Port 443
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

SyslogFacility AUTHPRIV
LogLevel INFO

LoginGraceTime 2m
PermitRootLogin no
StrictModes yes
MaxAuthTries 3
MaxSessions 1000

PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no

AllowUsers tclient
AllowTcpForwarding yes
X11Forwarding no
PermitTunnel no
AllowAgentForwarding no

ClientAliveInterval 30
ClientAliveCountMax 3
TCPKeepAlive yes
GatewayPorts clientspecified

Creating the Dedicated Systemd Server Service

Create /etc/systemd/system/sshd-remotesupport.service:

# /etc/systemd/system/sshd-remotesupport.service
[Unit]
Description=OpenSSH server daemon for Remote Support Tunnels
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.service
Wants=sshd-keygen.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D -f /etc/ssh/sshd_config_support $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

Enable and start the gateway service:

systemctl daemon-reload
systemctl enable sshd-remotesupport.service
systemctl start sshd-remotesupport.service

---

Step 2: Establishing AutoSSH Systemd Services on Remote Clients

On your target workstations or thin clients, install AutoSSH and configure a persistent systemd service to maintain an active reverse tunnel back to your gateway.

# Install AutoSSH package
yum install autossh -y

Manual Tunnel Verification

Verify the reverse tunnel manually by forwarding local SSH port 2748 to a dynamic port on the gateway:

autossh -M 0 -o "ServerAliveInterval 20" -o "ServerAliveCountMax 3" -R 0:localhost:2748 tclient@remotesupport.induslevel.com -i /home/tclient/remotesupport.pem -p 443 -o StrictHostKeyChecking=no

Creating the AutoSSH Systemd Client Service

To keep the tunnel running automatically in the background, create /etc/systemd/system/autossh-remotesupport.service:

# /etc/systemd/system/autossh-remotesupport.service
[Unit]
Description=AutoSSH reverse tunnel service for remote support
After=network.target

[Service]
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh -M 0 -T -o "ServerAliveInterval 20" -o "ServerAliveCountMax 3" -o "StrictHostKeyChecking=no" -o "UserKnownHostsFile=/dev/null" -NR 0:localhost:2748 tclient@remotesupport.induslevel.com -i /home/tclient/remotesupport.pem -p 443

[Install]
WantedBy=multi-user.target

Enable and start the AutoSSH client daemon:

systemctl daemon-reload
systemctl enable autossh-remotesupport.service
systemctl start autossh-remotesupport.service

---

Step 3: Discovering Dynamic Remote Ports (Helper Script)

When remote clients establish reverse tunnels using 0:localhost:2748, the gateway assigns a random high port dynamically. To identify the exact local port mapped to a remote workstation's public IP address, deploy this helper script (findsshport.sh) on your gateway server:

#!/bin/bash
# /usr/local/bin/findsshport.sh
publicip=$1
sshpid=$(netstat -anp | grep tclient | grep ESTABLISHED | grep $publicip | awk '{print $7}' | awk -F"/" '{print $1}')
childpid=$(ps -ef | grep tclient | grep -v root | grep $sshpid | awk '{print $2}')
netstat -anp | grep 127.0.0.1 | grep $childpid | awk '{print $4}' | awk -F: '{print $2}'

Execute the script passing the client's public IP address:

findsshport.sh 198.51.100.45

---

Step 4: Automated Fleet Administration via Ansible

With reverse tunnels active, you can administer the entire remote fleet simultaneously from your gateway server using Ansible.

1. Constructing Dynamic Ansible Inventory

Extract all active reverse tunnel ports listening on 127.0.0.1 and format them into an Ansible inventory file (hosts):

netstat -ntlp | grep tclient | grep 127.0.0.1 | awk '{print $4}' | awk -F: '{print $2" ansible_ssh_port="$2" ansible_ssh_host=127.0.0.1"}' > hosts

2. Executing Ad-Hoc Forensic Queries

Execute Ansible ad-hoc queries across all connected tunnels to inspect active background processes:

# Verify monitoring script execution across all connected thin clients
ansible --private-key=/root/tclient.key -u tclient --ssh-common-args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" -i hosts all -m shell -a 'ps -ef | grep monitor.sh | grep -v grep'

3. Executing Mass Maintenance Playbooks

Run automated maintenance playbooks across specific tunnels or the entire fleet:

# Execute playbook targeting specific dynamic port
ansible-playbook -v -i 10.10.240.93:2748, -u ansible -b --private-key=/home/mawan03/ansible-playbook/ssh-key/ansible_rsa /home/mawan03/ansible-playbook/remote-support/site.yml

# Execute playbook across all connected tunnels with specific tags
ansible-playbook -vv -i hosts -u ansible -b --private-key=/home/ec2-user/migration-WFH-cloud/ssh-key/ansible_rsa /home/ec2-user/migration-WFH-cloud/playbook/site.yml --tags "install_autossh"

References

• Permanent SSH Tunnels with AutoSSH
• SSH Tunneling for Fun and Profit