I was recently assigned the task of evaluating DynaTrace, an industry-leading application performance monitoring (APM) tool.
As part of the evaluation, I configured a three-node cluster along with one Environment ActiveGate and one Cluster ActiveGate node. Below is a step-by-step guide covering the installation and configuration process for this three-node DynaTrace Managed Cluster.
1. Hardware & VM Specifications
Based on the recommendations from DynaTrace for a Micro node size, I provisioned the virtual machines with slightly higher specs to ensure optimal performance under load:
- RAM: 64 GB
- vCPU: 8 Cores
- Disks: 2 separate disks configured as follows:
- Disk 1 (50 GB): LVM with XFS filesystem to allow for easy online expansion in the future.
/boot: 1 GBswap: 8 GB/: 41 GB
- Disk 2 (500 GB): LVM with XFS filesystem dedicated to Dynatrace binary and data storage.
/opt: 50 GB/var/opt/: 450 GB
- Disk 1 (50 GB): LVM with XFS filesystem to allow for easy online expansion in the future.
After virtual machine provisioning, the disk layout should look like this:
df -hOutput:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 9.0M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/rhel-root 41G 3.2G 38G 8% /
/dev/sda1 1014M 182M 833M 18% /boot
/dev/mapper/dt-opt 50G 4.2G 46G 9% /opt
/dev/mapper/dt-var_opt 442G 3.2G 439G 1% /var/opt
tmpfs 6.3G 0 6.3G 0% /run/user/0 2. Network & Hostname Configuration
The cluster consists of the following three nodes. ActiveGates will be added to these later.
| Hostname | IP Address |
|---|---|
lhr-dt01.induslevel.com | 172.16.50.101 |
lhr-dt02.induslevel.com | 172.16.50.110 |
lhr-dt03.induslevel.com | 172.16.50.122 |
3. Base System OS Preparation
Follow these steps to prepare the OS on all three virtual machines.
Register the Servers (RHEL only)
If you are using Red Hat Enterprise Linux, register the VMs with the Red Hat Subscription Portal to enable software repositories. You can skip this step if you are using CentOS 7.
subscription-manager register --username username@induslevel.com
subscription-manager attach --autoSystem Update
Update the package cache and upgrade existing packages:
yum update -yEnable EPEL Repository
The Extra Packages for Enterprise Linux (EPEL) repository provides useful sysadmin utilities:
rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpmInstall System Utilities & Helpers
Install common debugging and administration tools:
yum install -y wget mlocate telnet htop atop iftop open-vm-tools && updatedb4. Time Synchronization (NTP) Configuration
DynaTrace Managed Clusters require accurate time sync across all nodes. If time drift occurs, the installation will abort.
Install the NTP Daemon
yum install ntp -yConfigure NTP Servers
Update /etc/ntp.conf to use your internal NTP servers. (Skip this step or update with public servers if you do not have internal NTP servers):
sed -i '/^server 2/d' /etc/ntp.conf
sed -i '/^server 3/d' /etc/ntp.conf
sed -i 's/^server 0.*/server lhr-ntp1.induslevel.com iburst/g' /etc/ntp.conf
sed -i 's/^server 1.*/server lhr-ntp2.induslevel.com iburst/g' /etc/ntp.confReview NTP Configuration
Verify that your NTP configuration file contains the correct servers:
cat /etc/ntp.conf | egrep -v "^$|#"Output:
driftfile /var/lib/ntp/drift
restrict default nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict ::1
server lhr-ntp1.induslevel.com iburst
server lhr-ntp2.induslevel.com iburst
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
disable monitorForce Initial Sync and Start Daemon
Perform a one-time hard synchronization of the system clock and start the NTP service:
ntpdate -u lhr-ntp1.induslevel.com
systemctl enable ntpd && systemctl start ntpd && systemctl status ntpd && ntpq -c peersOutput:
ntpd.service - Network Time Service
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2021-09-11 21:06:09 PKT; 31min ago
Main PID: 1421 (ntpd)
Tasks: 1
CGroup: /system.slice/ntpd.service
+-1421 /usr/sbin/ntpd -u ntp:ntp -g
Sep 11 21:06:09 lhr-dt01.induslevel.com ntpd[1421]: Listen normally on 4 lo ::1 UDP 123
Sep 11 21:06:09 lhr-dt01.induslevel.com ntpd[1421]: Listen normally on 5 ens192 fe80::917d:c4a4:4545:8df2 UDP 123
Sep 11 21:06:09 lhr-dt01.induslevel.com ntpd[1421]: Listening on routing socket on fd #22 for interface updates
Sep 11 21:06:09 lhr-dt01.induslevel.com ntpd[1421]: 0.0.0.0 c016 06 restart
Sep 11 21:06:09 lhr-dt01.induslevel.com ntpd[1421]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
Sep 11 21:06:09 lhr-dt01.induslevel.com ntpd[1421]: 0.0.0.0 c011 01 freq_not_set
Sep 11 21:09:27 lhr-dt01.induslevel.com ntpd[1421]: 0.0.0.0 c614 04 freq_mode
Sep 11 21:24:43 lhr-dt01.induslevel.com ntpd[1421]: 0.0.0.0 0612 02 freq_set kernel 152.358 PPM
Sep 11 21:24:43 lhr-dt01.induslevel.com ntpd[1421]: 0.0.0.0 061c 0c clock_step +0.152256 s
Sep 11 21:24:44 lhr-dt01.induslevel.com ntpd[1421]: 0.0.0.0 c618 08 no_sys_peer
remote refid st t when poll reach delay offset jitter
==============================================================================
*11.13.164.111 129.250.35.251 3 u 5 64 377 1.425 -93.124 44.570
12.14.164.112 .STEP. 16 u - 512 0 0.000 0.000 0.0005. Firewall & Security Configuration
Firewall Settings
For testing and evaluation, I set the default zone to trusted. In production, the DynaTrace installer automatically manages and opens the specific ports it needs to operate.
firewall-cmd --set-default=trustedSELinux Settings
Disable SELinux in /etc/selinux/config:
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/configReboot the virtual machines for changes to take effect:
rebootRecommended Baseline Snapshot
This is an ideal moment to take a VM snapshot of all nodes. If anything goes wrong during the cluster setup, you can easily roll back to this clean baseline.
6. Download & Verify Installer
The managed server package is around 1.1 GB. Retrieve the script via a special download link provided by your DynaTrace account representative:
wget -O dynatrace-managed.sh "https://mcsvc.dynatrace.com/downloads/installer/get/latest?token=xxxxxxxxxxxxxxxxxxxxxxxxxxx"Once downloaded, check the cryptographic integrity of the installer:
wget -qO dt-root.cert.pem https://mcsvc.dynatrace.com/dt-root.cert.pem; wget -qO dynatrace-managed.sh.sig https://mcsvc.dynatrace.com/downloads/signature?filename=$(grep -am 1 'ARCH_FILE_NAME=' dynatrace-managed.sh | cut -d= -f2 |sed 's/.tar.gz$//'); openssl cms -inform PEM -binary -verify -CAfile dt-root.cert.pem -in dynatrace-managed.sh.sig -content dynatrace-managed.sh > /dev/nullYou can also perform a self-check using the installer script:
/bin/sh dynatrace-managed.sh --self-check7. Installing Node 1 (Primary Cluster Seed)
Initiate the cluster installation on the first node (lhr-dt01). You will need the license key supplied by Dynatrace Sales:
/bin/sh dynatrace-managed.sh --license XXXXXXXXXXXXXXXXProceed through the installation using the default paths:
Starting Dynatrace 1.224.84.20210902-165955 installer ... OK
To continue installation please accept our Terms of Use, which are available
at https://www.dynatrace.com/company/legal/customers/
By submitting 'Agree' you are confirming that you have read, understood, and
accept to the Terms of Use. Otherwise, press Ctrl+C to cancel installation.
? Agree
Verifying system compatibility ... OK
Verifying RAM size ... OK
Type the full path to your directory for Dynatrace binaries
[/opt/dynatrace-managed]?
Type the full path to your directory for Dynatrace data
[/var/opt/dynatrace-managed]?
Do you want to keep all Dynatrace data in this directory?
[y]?
Do you want to join an existing Dynatrace cluster (y/n)?
[n]?
Enter the command which should be used for executing commands with
superuser privileges. This command should contain variable $CMD.
[sudo -n $CMD]?
Verifying disk space ... OK
Testing connection to Dynatrace Mission Control ... OK
Verifying system connectivity ... OK
Preparing system user for Dynatrace ... OK
Initializing installation ... OK
Checking user permissions ... OK
Downloading Dynatrace OneAgent. This may take a few minutes ...
Installing. This may take a few minutes ... OK
Fixing selinux rules for binaries if needed ... Skipped
Installing Nodekeeper ... OK
Setting up cluster configuration. This may take a few minutes ... OK
Starting Dynatrace. This may take up to half an hour ... OK
Configuring Dynatrace. This may take a few minutes ... OK
Installation completed successfully after 11 minutes 55 seconds.
Dynatrace binaries are located in directory /opt/dynatrace-managed
Dynatrace data is located in directory /var/opt/dynatrace-managed
Dynatrace metrics repository is located in directory /var/opt/dynatrace-managed/cassandra
Dynatrace Elasticsearch store is located in directory /var/opt/dynatrace-managed/elasticsearch
Dynatrace server store is located in directory /var/opt/dynatrace-managed/server
Dynatrace session replay store is located in directory /var/opt/dynatrace-managed/server/replayData
You can now log into your Dynatrace Server at https://172.16.50.1018. Adding Node 2 to the Cluster
Open a browser and navigate to your first node's Cluster Management Console (CMC) address:
https://172.16.50.101/cmc#cm/nodeDownload;gf=all
From here, copy the generated download link and extension command for the second node.
Download Installer on Node 2
Run this command on lhr-dt02 to pull the installer bundle directly from the seed node:
wget -O managed-installer.sh https://ghbxxx.dynatrace-managed.com/nodeinstaller/dt0c01.XXXXXXXXXXXXXXXXXXXXXXXXXXInstall and Register Node 2
Run the installer script, passing the --seed-auth token obtained from your CMC:
/bin/sh managed-installer.sh --seed-auth dt0c01.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXFollow the system prompts to extend the cluster:
Starting Dynatrace 1.224.84.20210902-165955 installer ... OK
Verifying system compatibility ... OK
Verifying RAM size ... OK
Type the full path to your directory for Dynatrace binaries
[/opt/dynatrace-managed]?
Type the full path to your directory for Dynatrace data
[/var/opt/dynatrace-managed]?
Do you want to keep all Dynatrace data in this directory?
[y]?
Enter the command which should be used for executing commands with
superuser privileges. This command should contain variable $CMD.
[sudo -n $CMD]?
Verifying disk space ... OK
Testing connection to Dynatrace Mission Control ... OK
Verifying system connectivity ... OK
Collecting information about Dynatrace cluster ... OK
Preparing Dynatrace cluster for extension ...
Waiting 6 minutes for permission to add new node ...
Permission status: pre-check pending
Checking again in 20 seconds (5 minutes 59 seconds left)
Permission status: Join is possible
Permission is granted
Adding this node to Dynatrace cluster ... OK
Preparing system user for Dynatrace ... OK
Extending existing cluster (seed node IP: 172.16.50.101) with new node.
Initializing installation ... OK
Checking user permissions ... OK
Downloading Dynatrace OneAgent. This may take a few minutes ... OK
Installing. This may take a few minutes ... OK
Fixing selinux rules for binaries if needed ... Skipped
Installing Nodekeeper ... OK
Setting up cluster configuration. This may take a few minutes ... OK
Starting Dynatrace. This may take up to half an hour ... OK
Configuring Dynatrace. This may take a few minutes ... OK
Installation completed successfully after 13 minutes 21 seconds.
Dynatrace binaries are located in directory /opt/dynatrace-managed
Dynatrace data is located in directory /var/opt/dynatrace-managed
Dynatrace metrics repository is located in directory /var/opt/dynatrace-managed/cassandra
Dynatrace Elasticsearch store is located in directory /var/opt/dynatrace-managed/elasticsearch
Dynatrace server store is located in directory /var/opt/dynatrace-managed/server
Dynatrace session replay store is located in directory /var/opt/dynatrace-managed/server/replayData
You can now log into your Dynatrace Server at https://172.16.50.1109. Adding Node 3 to the Cluster
Repeat the extension process for the final node (lhr-dt03).
Download Installer on Node 3
wget -O managed-installer.sh https://ghbxxx.dynatrace-managed.com/nodeinstaller/dt0c02.XXXXXXXXXXXXXXXXXXXXXXXXXXInstall and Register Node 3
Run the installer script with the token for the third node:
/bin/sh managed-installer.sh --seed-auth dt0c02.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXThe output will be identical to Node 2's extension process, confirming the successful extension of your Cassandra metrics and Elasticsearch store across all three nodes.
10. Verifying Cluster Health
Your 3-node Dynatrace Managed Cluster is now fully operational!
You can access the Cluster Management Console (CMC) by navigating to the IP address of any of the three nodes:
https://172.16.50.101https://172.16.50.110https://172.16.50.122
For high availability and ease of access in production environments, you should configure a DNS A-record (e.g., dynatrace.induslevel.com) with all three server IP addresses to allow for seamless round-robin load balancing.
