Blog Projects Uses About
Back to Blog

Automating Proxmox VM Lifecycle with Ansible and Semaphore

• 7 min read
ansible proxmox automation semaphore

Manually SSH-ing into each of my 18 hosts to apply updates, create users, or change configurations doesn’t scale — even in a homelab. Ansible handles the automation, and Semaphore provides a web UI that makes running playbooks as simple as clicking a button.

Why Semaphore?

Ansible on its own is powerful but requires command-line access, inventory management, and remembering the right flags for each playbook. Semaphore adds:

  • Web-based UI: Run playbooks from a browser — no terminal needed
  • Task History: See what ran, when, and whether it succeeded
  • Scheduled Runs: Trigger playbooks on a cron schedule
  • Access Control: Grant team members playbook access without SSH keys
  • Inventory Management: Visual inventory editor instead of flat files

For a homelab, it turns Ansible from a power tool into an appliance.

Architecture

Semaphore runs on a dedicated host and stores its playbooks, inventory, and credentials in a local database. The playbooks themselves live in a Git repository, pulled automatically when a task runs.

Semaphore Web UI → Ansible Engine → SSH → Target Hosts

All Ansible commands execute from Semaphore’s working directory at semaphore/opt/semaphore/tmp/proxmox/, which contains the playbooks, roles, and inventory files.

The Playbook Library

VM Creation: pve_create_vm.yml

Creating a new VM in Proxmox involves dozens of API calls — specifying CPU, memory, disk, network, cloud-init parameters, and more. This playbook automates the entire process:

  • Provisions a VM on the specified Proxmox node
  • Configures hardware resources (CPU, RAM, disk)
  • Attaches to the correct VLAN/network
  • Applies cloud-init for initial OS configuration
  • Starts the VM and waits for SSH availability

What used to take 15 minutes of clicking through the Proxmox UI now takes 30 seconds.

VM Onboarding: pve_onboard.yml

After a VM is created, it needs to be prepared for Ansible management. The onboarding playbook handles:

  1. Creates the ansible user with SSH key authentication
  2. Configures passwordless sudo for the ansible user
  3. Deploys SSH authorized keys from a central key store
  4. Sets the hostname to match the inventory name
  5. Configures basic networking (DNS, NTP)

This is the one playbook I run manually the first time (since the new VM doesn’t have the ansible user yet). After onboarding, all subsequent automation uses the ansible user.

Common Linux Configuration: doug-common-linux-playbook.yml

This is the workhorse playbook that applies my standard Linux baseline. It uses a custom role called doug-common-linux with modular task files:

Date/NTP Configuration:

  • Sets timezone to America/New_York
  • Configures NTP synchronization
  • Ensures consistent timestamps across all hosts

Hostname Management:

  • Sets hostname from inventory
  • Updates /etc/hosts with the correct hostname mapping

User Management:

  • Creates standard user accounts
  • Configures SSH keys
  • Sets password policies
  • Manages sudo access

Package Installation:

  • Installs a standard package set (htop, curl, wget, jq, etc.)
  • Removes unnecessary default packages
  • Configures unattended security updates

Mail Configuration:

  • Configures outbound mail relay
  • Sets up mail aliases for system notifications

System Configuration:

  • Kernel parameter tuning
  • Sysctl settings for network performance
  • Log rotation configuration

System Updates: update_linux_servers.yml

Rolling updates across all managed hosts:

ansible-playbook -i inventory update_linux_servers.yml

This playbook:

  • Runs apt update && apt upgrade on Debian-based hosts
  • Handles package holds and pinned versions
  • Reports which packages were updated
  • Optionally reboots hosts that require it (kernel updates)
  • Processes hosts in serial to avoid taking down all services simultaneously

Running this through Semaphore means I can trigger updates from my phone, watch the progress in real-time, and review the results later — all without opening a terminal.

Additional Playbooks

  • install_alloy.yml: Deploys Grafana Alloy agent for metrics collection
  • install-nvim.yml: Installs and configures Neovim with my preferred settings
  • ping.yml: Simple connectivity test across all hosts (great for verifying after network changes)

The Enigma Secrets Manager

Ansible playbooks often need credentials — root passwords, API tokens, service accounts. Hardcoding these in playbooks or inventory files is a security risk.

I use a custom Ansible lookup plugin called Enigma that reads credentials from ~/.enigma.json:

# In a playbook
common_users_root_password: "{{ lookup('enigma', '7604', 'password') }}"

Enigma stores credentials as a JSON key-value store with numeric IDs. Each entry can have arbitrary fields (username, password, bot_token, chat_id, etc.). The file lives outside the Git repository, so credentials never end up in version control.

This pattern keeps secrets management simple — no HashiCorp Vault server to maintain, no cloud secrets manager to pay for. Just a JSON file with restrictive permissions on the Semaphore host.

Custom Roles

doug-common-linux

This is my primary Ansible role, organized into modular task files:

roles/doug-common-linux/
├── tasks/
│   ├── main.yml          # Imports all task files
│   ├── date-ntp.yml      # Timezone and NTP
│   ├── hostname.yml      # Hostname configuration
│   ├── users.yml         # User management
│   ├── system.yml        # System configuration
│   ├── packages.yml      # Package management
│   ├── mail.yml          # Mail relay setup
│   └── domain.yml        # Domain joining (optional)
├── defaults/
│   └── main.yml          # Default variables
├── handlers/
│   └── main.yml          # Service restart handlers
└── templates/
    └── ...               # Configuration file templates

Each task file is self-contained and can be skipped with tags. Need to update just the NTP configuration across all hosts? Run the playbook with --tags ntp and only that task file executes.

Workflow Example: Adding a New Host

Here’s the typical workflow when I spin up a new LXC container or VM:

  1. Create the VM/Container (Proxmox UI or pve_create_vm.yml)
  2. Onboard — Run pve_onboard.yml to create the ansible user and configure SSH
  3. Configure — Run doug-common-linux-playbook.yml to apply the standard baseline
  4. Deploy Service — Install the specific service (Docker, Zabbix Agent, etc.)
  5. Monitor — Add to Zabbix monitoring (usually automatic via discovery)

Steps 2–4 are all Semaphore button clicks. A new host goes from bare OS to fully configured in about 10 minutes, with consistent configuration every time.

Lessons Learned

1. Idempotency Is Everything

Every task in every playbook must be idempotent — running it twice produces the same result as running it once. This means using state: present instead of shell commands, checking before modifying, and avoiding destructive operations.

2. Serial Execution for Updates

Never run updates on all hosts simultaneously. If an update breaks something, you want to catch it on the first host before it propagates. Set serial: 1 or serial: 3 in update playbooks.

3. Test in Dev, Deploy in Prod

I have template VMs specifically for testing playbook changes. New role modifications get tested on a throwaway VM before running against production hosts. Semaphore’s task history makes it easy to compare test runs against production runs.

4. Keep Roles Modular

The doug-common-linux role started as a monolithic task file. Breaking it into separate files for users, packages, NTP, etc. made it dramatically easier to maintain and debug. Each file is small enough to understand at a glance.

5. Git-Tracked Playbooks

All playbooks and roles live in a Git repository. Semaphore pulls from Git on each run, so the latest changes are always applied. This also provides full audit history — who changed what, when, and why.

What’s Next

  • Expanded Zabbix Agent deployment: Automate Zabbix Agent 2 installation and PSK configuration as part of the onboarding playbook
  • Backup verification: Playbook to verify Proxmox Backup Server integrity and test restores
  • Certificate rotation: Automate Let’s Encrypt certificate renewal tracking

For more on the monitoring that works alongside this automation, see Monitoring Everything: Zabbix 7 in a Homelab. For the complete infrastructure overview, check out Building My Homelab.