Keep backups and recovery plans for Proxmox upgrades

I upgraded my personal Proxmox node and found out the hard way that extra packages on the hypervisor can ruin an otherwise routine upgrade. I lost a pfSense box because Network UPS Tools (NUT) rebooted the UPS during a Debian/Proxmox upgrade. Proxmox itself came back after I booted an older kernel and finished package configuration, but the mess still cost time and hardware.

What broke

The usual signs were all there:

• apt or dpkg stalled or threw errors such as dpkg: error processing package <package-name> (--configure): and subprocess installed post-installation script returned error exit status 1.
• systemctl status nut-server.service showed Active: failed (Result: exit-code).
• The package manager complained about held or broken packages, including E: Unmet dependencies. Try 'apt --fix-broken install'.
• The upgrade stopped at package configuration, usually after Setting up ....
• Services dropped during the upgrade, and hosts on the network started shutting down.
• The host booted an older kernel after the kernel change.
• The UPS did a controlled shutdown from the hypervisor, and logs showed NUT driving it.

Where it went wrong

The problem was the post-install work. Packages run their postinst scripts during dpkg --configure, and a package that controls hardware can kick off actions at that point. Debian packages on PVE can also touch systemd services and kernel modules, which is normal enough until a package starts managing power.

In my case, NUT Tools on the hypervisor triggered the UPS during the upgrade. That cascade took out connected devices too. A UPS reboot means a power cycle across AC-switched loads, and some hardware does not enjoy that at all.

How I found it

The logs made it obvious once I went looking.

• Check the apt logs:

/var/log/apt/history.log
/var/log/dpkg.log

Look for postinst entries and errors around the upgrade time.

• Check systemd and the journal:

journalctl -b -1 --no-pager | grep -i nut
journalctl -u nut-server.service

I was looking for lines such as upsmon: shutdown or received signal SIGTERM.

• Check what owns the UPS on the host:

dpkg -l | grep -i nut

The expectation was no NUT packages on the hypervisor, or at least just nut-server. What I actually had was both nut-server and nut-client.

• Check package state:

dpkg --configure -a

If that hangs or throws a post-installation error, the package script is part of the problem.

• Check the service state:

systemctl status nut-server.service

That was failed on the broken node.

• List non-Proxmox packages:

apt list --installed | grep -v pve

Then inspect the postinst scripts for anything that calls upsmon, upscmd, or system power actions:

ls -l /var/lib/dpkg/info/<package>.postinst
cat /var/lib/dpkg/info/<package>.postinst

Getting the node back

1. Boot the host into an older kernel from the GRUB menu. That can get around a kernel-package mismatch.
2. Finish package configuration:

sudo dpkg --configure -a
sudo apt --fix-broken install

If a script keeps firing the same dangerous action, stop there and edit the script.

3. If the package’s postinst is unsafe, edit it for the recovery run:

sudo nano /var/lib/dpkg/info/<package>.postinst

Comment out the lines that invoke hardware control, then put the file back afterwards.

4. Run dpkg --configure -a again.

After that, remove or purge the offending packages from the hypervisor:

sudo apt remove --purge nut-server nut-client

If I need UPS management at all, it stays off the hypervisor. A dedicated appliance or a properly isolated VM is less likely to drag the whole host down with it.

What I check now

• Reboot the host into the newest kernel once package configuration is clean.
• Check service health with:

pveversion -v
systemctl list-units --type=service --state=failed

• Before a real upgrade, simulate one on a test host:

apt update && apt upgrade --simulate

I also restore a VM from backup to check the backup actually boots and the network comes up. Backups that only exist in theory are not much use when the host is already sulking.

• Before each Proxmox upgrade, list installed packages and flag anything non-standard:

apt list --installed | grep -vE 'proxmox|pve|pve-manager|pve-kernel'

My pre-upgrade checklist is simple: back up configs and VMs, drain anything non-essential from the hypervisor, and move hardware-control packages off the host.

For this sort of upgrade, keep the host minimal. Check every package that touches hardware, keep recovery steps written down, and do not assume a package update is harmless just because it came through apt.