Back to Blog

Fixing LXC Network Disconnects After Proxmox Apt Upgrades

6 min read

After a routine apt upgrade on one of my Proxmox nodes, I noticed that several LXC containers had silently lost network connectivity. The containers were running fine - services were up, processes were healthy - but they could not reach anything outside the container. No internet, no other hosts, nothing. The root cause turned out to be a Proxmox networking quirk that affects anyone running LXC containers, and the permanent fix is a small shell hook.

The Symptom#

The pattern is consistent:

  1. Run apt upgrade on a Proxmox node
  2. Upgrade completes without errors
  3. Some or all LXC containers on that node lose network connectivity
  4. ip addr inside the container shows the interface up with an IP address
  5. ping 10.0.1.1 from inside the container fails silently (no response)
  6. ping to the container's IP from another host also fails

The containers look healthy from the Proxmox UI. Their status shows "running" and the console is accessible. But network traffic simply stops flowing.

The Root Cause#

When Proxmox's networking service restarts during an apt upgrade - triggered by kernel updates, network-related package upgrades, or systemd restarts - it tears down and rebuilds the Linux bridge (vmbr0). The problem is that it does not re-attach the LXC container veth interfaces to the bridge.

Proxmox uses virtual Ethernet pairs for LXC container networking. Each container has a veth device inside the container namespace and a corresponding veth on the host. Both ends are created when the container starts. The host end is attached to vmbr0 so the container can communicate with the network.

When vmbr0 is torn down and rebuilt, the bridge comes back up clean - but the existing veth interfaces that were previously attached are now dangling. They still exist as Linux interfaces on the host, but they are no longer bridged to anything.

You can verify this by running on the Proxmox host:

bridge link show vmbr0

On a working node, every running container's veth should appear in the output. After an upgrade, you will see gaps - missing veth interfaces for affected containers.

To see the orphaned veths:

ip link show | grep veth

Compare this against what bridge link show vmbr0 reports. Any veth present in ip link show but not in bridge link show vmbr0 is detached.

The Manual Fix#

To restore connectivity for an affected container, re-attach its veth to the bridge:

brctl addif vmbr0 vethXXXXXX

Where vethXXXXXX is the orphaned veth interface name. You can find the mapping between veth names and container IDs in the Proxmox network logs or by checking:

cat /proc/net/dev | grep veth

And cross-referencing against the container config in /etc/pve/lxc/<ctid>.conf.

This works immediately - no container restart required. But it is a manual fix that needs to be applied after every problematic upgrade.

The Permanent Fix#

The real solution is an /etc/network/if-up.d/ hook that fires whenever a network interface comes up. When vmbr0 comes back up after a restart, the hook iterates through all running LXC containers and re-attaches any detached veth interfaces.

Create /etc/network/if-up.d/reattach-lxc-veths:

#!/bin/bash
# Re-attach LXC veth interfaces to vmbr0 after bridge restart
 
BRIDGE="vmbr0"
 
# Only run when vmbr0 comes up
if [ "$IFACE" != "$BRIDGE" ]; then
    exit 0
fi
 
# Wait briefly for the bridge to fully initialize
sleep 2
 
# Find all running LXC containers
for ctid in $(pct list 2>/dev/null | awk 'NR>1 && $2=="running" {print $1}'); do
    # Get the veth name for this container
    veth=$(grep -oP 'veth\S+' /proc/net/dev 2>/dev/null | while read v; do
        if cat "/sys/class/net/$v/ifindex" 2>/dev/null | grep -q .; then
            echo "$v"
        fi
    done | head -1)
 
    # Check if the veth for this container is already in the bridge
    for iface in $(ls /sys/class/net/ | grep veth); do
        # Check if this veth belongs to this container
        if ip link show "$iface" 2>/dev/null | grep -q "veth"; then
            # Check if it is attached to the bridge
            if ! bridge link show "$BRIDGE" 2>/dev/null | grep -q "$iface"; then
                # Re-attach the orphaned veth
                brctl addif "$BRIDGE" "$iface" 2>/dev/null && \
                    logger "reattach-lxc-veths: re-attached $iface to $BRIDGE"
            fi
        fi
    done
done

Make it executable:

chmod +x /etc/network/if-up.d/reattach-lxc-veths

Deploy this to all Proxmox nodes. On my cluster that means pve01, pve02, and pve03 each get the hook via Ansible:

- name: Deploy veth reattach hook
  copy:
    src: files/reattach-lxc-veths
    dest: /etc/network/if-up.d/reattach-lxc-veths
    mode: '0755'
    owner: root
    group: root

Verifying the Fix#

After deploying the hook, you can test it without waiting for an actual upgrade by manually restarting the networking service:

systemctl restart networking

Watch the system log for the hook output:

journalctl -f | grep reattach-lxc-veths

You should see entries like:

reattach-lxc-veths: re-attached veth100i0 to vmbr0
reattach-lxc-veths: re-attached veth101i0 to vmbr0

Then verify containers still have connectivity:

pct exec 100 -- ping -c 3 10.0.1.1

Per-Node Veth Mapping#

For reference, here is the veth-to-container mapping on my cluster after mapping it out once everything was stable. This is useful for manual debugging when the hook is not yet deployed or fails:

NodeCTIDServiceVeth
pve01100traefikveth100i0
pve01101homepageveth101i0
pve01113plexveth113i0
pve02114ollamaveth114i0
pve02115zabbixveth115i0
pve03120docker01veth120i0

Your veth names will differ, but the pattern is consistent: veth<ctid>i<net_index>.

Why Not Just Restart the Containers?#

The obvious workaround is to restart all LXC containers after an upgrade. This works, but it has real costs:

  • Service interruption: Plex, n8n, and other services stop and restart, interrupting any active sessions
  • Database integrity: Some services (PostgreSQL, MySQL) do not like abrupt restarts during active connections
  • Downtime window: With 15+ containers across three nodes, a rolling restart takes meaningful time

The veth re-attachment approach has zero service interruption. The containers keep running; the network path is just restored without the container ever knowing it was broken.

The Bigger Pattern#

This issue is one of several places where Proxmox's upgrade path and LXC container management can interact in unexpected ways. The general principle: after any Proxmox apt upgrade that touches networking, kernel, or systemd packages, verify container connectivity before assuming everything is fine.

I added a Zabbix check that runs 15 minutes after a Semaphore update task completes, verifying that all monitored hosts are still reachable. If the check fails, Telegram gets an alert before I would otherwise notice.

For details on the Ansible automation that deploys this fix across the cluster, see Automating Proxmox with Ansible and Semaphore. For the monitoring that catches when it goes wrong anyway, check out Monitoring Everything: Zabbix 7 in a Homelab.

Related Posts