Zabbix PSK Encryption: Why Ansible Was Breaking It on Every Run

Zabbix Agent PSK encryption was working fine after initial setup. Then I automated the agent deployment with Ansible and Semaphore - and every time the playbook ran, a handful of agents would silently lose their encrypted connection. No errors, no alerts, just passive checks quietly switching to unencrypted mode. It took a while to track down because the failure mode was subtle, but the root cause turned out to be one of Ansible's most confusing lookup plugin behaviors.

Background: How PSK Works in Zabbix#

Pre-Shared Key (PSK) encryption in Zabbix requires two things to match between the agent and the server:

PSK Identity: A string identifying the key (I use the hostname, e.g., docker01)
PSK Value: A 128-character hex string used as the actual key

Both the agent config on the host (/etc/zabbix/zabbix_agent2.conf) and the host definition in Zabbix Server must have the same PSK value. If they diverge, the agent falls back to unencrypted connections (if TLSAccept=psk,unencrypted) or fails entirely (if TLSAccept=psk only).

My agent config includes:

TLSConnect=psk
TLSAccept=psk,unencrypted
TLSPSKIdentity=docker01
TLSPSKFile=/etc/zabbix/zabbix_agent2.psk

And the PSK value in /etc/zabbix/zabbix_agent2.psk is a 128-character hex string generated once with:

openssl rand -hex 64

The Symptom#

After running the doug-common-linux playbook through Semaphore, Zabbix would show some agents connecting with TLS and others suddenly showing as unencrypted. The pattern was inconsistent - sometimes one host, sometimes three, sometimes none.

Checking the Zabbix server log:

Zabbix agent item "..." was not processed: Cannot connect to IPADDR:10050: connection of type PSK is not configured

And on the agent side:

Failed to connect to server: [ssl_connect] failed to connect using PSK

The PSK file existed on the host. The Zabbix server still had a PSK configured. But they were different values.

Root Cause: Ansible's `password` Lookup#

My Ansible role used the Ansible password lookup plugin to generate and store PSK values:

- name: Set Zabbix PSK
  set_fact:
    zabbix_psk: "{{ lookup('password', '/tmp/zabbix-psk-{{ inventory_hostname }} chars=hexdigits length=128') }}"

The password lookup plugin is designed to generate a random value once and cache it on disk. The idea is that it generates the password on first run and reuses it on subsequent runs - making playbooks idempotent for password management.

The problem: the path /tmp/zabbix-psk-<hostname> is on the Semaphore host's filesystem, and /tmp is volatile. On every Semaphore container restart, system reboot, or tmpwatch cleanup, those files disappear. Next time the playbook runs, the password lookup finds no file, generates a new random value, writes it to /tmp, and proceeds to update the agent's PSK file with the new value - while leaving the old PSK value configured in Zabbix Server.

The result: agent PSK and server PSK are now different. Encrypted connections fail.

This is worse than it sounds because /tmp cleanup is silent. The playbook shows green across the board (task "Set Zabbix PSK" runs idempotently from Ansible's perspective), and the PSK file on the agent actually gets updated. It is only when Zabbix tries to connect that you discover the mismatch.

The Fix: Persistent PSK Storage#

The solution is to store the generated PSK files in a persistent location on the Semaphore host - somewhere that survives reboots and cleanup jobs:

- name: Set Zabbix PSK
  set_fact:
    zabbix_psk: "{{ lookup('password', '/opt/semaphore/psk/zabbix-psk-{{ inventory_hostname }} chars=hexdigits length=128') }}"

Create the directory on the Semaphore host and set appropriate permissions:

mkdir -p /opt/semaphore/psk
chmod 700 /opt/semaphore/psk
chown semaphore:semaphore /opt/semaphore/psk

With PSK files stored in /opt/semaphore/psk/, they persist across:

Semaphore service restarts
OS reboots
/tmp cleanup jobs
Docker container restarts (if the directory is volume-mounted)

On the first run with the new path, Ansible generates new PSK values (the old /tmp files are gone anyway) and updates both the agent config and Zabbix Server. From that point on, the values are stable.

Updating Zabbix Server Automatically#

The PSK fix is only complete if Ansible also updates the host's PSK in Zabbix Server when it changes. I use the community.zabbix Ansible collection for this:

- name: Update PSK in Zabbix Server
  community.zabbix.zabbix_host:
    server_url: "http://10.0.1.103"
    login_user: "{{ zabbix_api_user }}"
    login_password: "{{ zabbix_api_password }}"
    host_name: "{{ inventory_hostname }}"
    tls_connect: 2  # PSK
    tls_accept: 3   # PSK + unencrypted
    tls_psk_identity: "{{ inventory_hostname }}"
    tls_psk: "{{ zabbix_psk }}"

This runs after the agent config is deployed. Zabbix API, agent PSK file, and Semaphore's stored value all update atomically. If the playbook is interrupted and only partially applied, re-running it will fix the inconsistency.

Additional Complication: TLS 1.3 and Go#

While debugging the PSK issue, I also discovered that some hosts - specifically the Proxmox nodes (pve01, pve02, pve03) and the Unraid NAS (Tower) - were failing encrypted connections for a different reason.

The Zabbix Agent 2 binary on these hosts is compiled with Go. Newer versions of OpenSSL (3.5.x) changed the TLS 1.3 handshake behavior in a way that breaks the TLS_PSK_WITH_AES_128_GCM_SHA256 cipher suite when one end is Go and the other is OpenSSL. The specific failure is in the TLS binder computation.

The affected hosts were upgraded to Debian Bookworm, which includes OpenSSL 3.5.x. The Zabbix Agent 2 binary uses Go's TLS stack, which apparently does not handle the new binder behavior correctly with some OpenSSL versions.

The workaround for these hosts: move them to unencrypted Zabbix agent connections. Since they are Proxmox hypervisors on a trusted internal network, and the monitoring data is not sensitive, this is acceptable. These hosts use SNMP for most monitoring anyway - Zabbix Agent is secondary.

# On pve01/pve02/pve03 and Tower
TLSConnect=unencrypted
TLSAccept=unencrypted

Longer term, the right fix is either waiting for the Go TLS library to be updated in Zabbix Agent 2, or switching these hosts to SNMP-only monitoring and removing the agent entirely.

pihole2 ARM64 Agent Upgrade#

The pihole2 host (Raspberry Pi, ARM64) was running Zabbix Agent 6.0.14 from the Raspbian repository - the only version available for ARM64 at the time I deployed it. After fixing the PSK issues on all other hosts, I upgraded pihole2 to Agent 7.0.27 which became available through the Raspbian repo.

The upgrade required adding the Zabbix 7 Raspbian repository:

wget -O /tmp/zabbix-release.deb https://repo.zabbix.com/zabbix/7.0/raspbian/pool/main/z/zabbix-release/zabbix-release_latest+raspbian12_armhf.deb
dpkg -i /tmp/zabbix-release.deb
apt update
apt install zabbix-agent2

With Agent 7.x on pihole2, active checks now work correctly (previously disabled due to version mismatch with Server 7.x), and PSK encryption can be enabled consistently.

Lessons Learned#

1. Never Use /tmp for Ansible Password Lookups#

The password lookup's file path needs to be persistent. /tmp seems convenient but it is a reliability trap. Use a dedicated directory under /var, /opt, or the application's data directory.

2. Test Idempotency Across Restarts#

An idempotent playbook should produce the same result whether it runs today or six months from now. Testing only on consecutive runs does not catch persistence bugs. Simulate a reboot or clean /tmp and run again.

3. Watch for Silent Fallback#

Zabbix's TLSAccept=psk,unencrypted setting means PSK failures are silent from a service perspective - the agent just falls back to unencrypted. Always monitor whether your agents are actually using PSK, not just whether they are reachable. The Zabbix web UI shows the TLS status per host in the Hosts section.

4. Know Your Go TLS Compatibility#

Go's TLS implementation and OpenSSL do not always agree on cipher suite behavior, especially across major version bumps. If encrypted connections start failing after a system upgrade, check whether the Go/OpenSSL combination is the culprit before blaming your configuration.

For the overall Zabbix monitoring architecture this PSK setup protects, see Monitoring Everything: Zabbix 7 in a Homelab. For the Ansible automation that triggered this bug, check out Automating Proxmox with Ansible and Semaphore.