Avoiding Upgrade Chaos: Best Practices for Proxmox Package Management

I upgraded my personal Proxmox node and learned the hard way that extra packages on the hypervisor can ruin an otherwise routine Proxmox upgrade. I lost a pfSense box because Network UPS Tools (NUT) rebooted the UPS during a Debian/Proxmox upgrade. Proxmox itself recovered after I booted an older kernel and finished package configuration, but the fallout cost time and hardware. This guide shows what to spot, how to find the root cause, and how to recover without guesswork.

What you see

Common error messages

• apt/dpkg stalls or throws errors such as:
  • dpkg: error processing package (–configure):
  • subprocess installed post-installation script returned error exit status 1
• systemctl shows services failing:
  • systemctl status nut-server.service → Active: failed (Result: exit-code)
• package manager complains about held or broken packages:
  • E: Unmet dependencies. Try ‘apt –fix-broken install’

Signs of upgrade failure

• Upgrade halts at package configuration. The apt output pauses after “Setting up …” or shows repeated retries.
• Multiple services stop during the upgrade. Hosts on the network go into shutdown cycles.
• VMs fail to restart after kernel change. Host boots into an older kernel automatically.

Unexpected system behaviour

• UPS controlled shutdowns originating from the hypervisor. Logs show NUT initiated a shutdown sequence on power events.
• Dependent devices (routers, routers on bare metal) fail to come back up. Hardware that was gracefully shut down can still fail to boot.

Where it happens

During Proxmox upgrades

• The risky bit is package postinst scripts. They run as part of dpkg –configure. A package that manages hardware can trigger actions at that stage.
• Upgrading Debian packages on PVE can touch systemd services and kernel modules. That is normal. It becomes dangerous when a package controls power.

Impact on connected devices

• A UPS reboot causes a power cycle across AC-switched loads. That forces graceful shutdowns, then starts a boot sequence.
• Some devices do not survive repeated power-cycles. A router’s flash or PSU can fail.

Specific scenarios with NUT Tools

• NUT may send shutdown commands when it detects a start/stop event from its local drivers or during service restart.
• If the hypervisor runs NUT and manages a networked UPS, a post-install hook may trigger the UPS. That can cascade into a network-wide shutdown.

Find the cause

Analysing logs

• Check the apt logs:
  • /var/log/apt/history.log
  • /var/log/dpkg.log
    Look for postinst entries and errors around the time of the upgrade.
• Check systemd and journal:
  • journalctl -b -1 –no-pager | grep -i nut
  • journalctl -u nut-server.service
    Look for lines such as “upsmon: shutdown” or “received signal SIGTERM”.

Diagnostic commands

• Who owns the UPS on this host:
  • dpkg -l | grep -i nut
  • Expected: no NUT packages on hypervisor, or nut-server present
  • Actual: nut-server and nut-client both installed
• Package status and failures:
  • dpkg –configure -a
  • Expected: completes with “Setting up … done”
  • Actual: hangs or throws “post-installation script returned error”
• Service status:
  • systemctl status nut-server.service
  • Expected: Active: active (running)
  • Actual: Active: failed (Result: exit-code)

Identifying problematic packages

• List packages that are not from the Proxmox repo:
  • apt list –installed | grep -v pve
• Inspect postinst scripts:
  • ls -l /var/lib/dpkg/info/.postinst
  • cat /var/lib/dpkg/info/.postinst
    Look for commands that call upsmon, upscmd, or system power actions.

Root cause (typical)

• A non-core package with hardware control runs a post-install or restart hook during the upgrade. That package triggers the UPS or other hardware, causing a cascade. On my node, NUT Tools did exactly that during a Debian/Proxmox upgrade.

Fix

Steps to recover Proxmox

1. Boot the host into an older kernel from the GRUB menu. That can avoid a kernel-package mismatch.
2. Finish package configuration:
   • sudo dpkg –configure -a
   • sudo apt –fix-broken install
     Expected outcome: dpkg runs through postinst scripts and completes. If a script repeats the dangerous action, stop and edit the script.
3. If a package’s postinst is unsafe, edit it temporarily:
   • sudo nano /var/lib/dpkg/info/.postinst
   • Comment out the lines that invoke hardware control. Mark the file with a comment so you remember to restore it.
4. Re-run dpkg –configure -a.

Reconfiguring packages

• Remove or purge offending packages from the hypervisor:
  • sudo apt remove –purge nut-server nut-client
• Move UPS management off the hypervisor to a dedicated appliance or a VM with proper isolation.
• If you need NUT on the hypervisor, configure it to not trigger shutdowns during upgrades. Use config parameters like NOTIFYFLAG or disable upsmon during package operations.

Alternatives for UPS management

• Run UPS management on an LXC or VM that does not hold control of host power.
• Use a networked UPS device model that supports SNMP monitoring only, leaving shutdown control to a resilient device.
• Keep a spare router and spare hardware tested for quick swap if a dependent device dies.

Check it’s fixed

Verifying system stability

• Reboot the host into the newest kernel after packages configure cleanly.
• Check service health:
  • pveversion -v
  • systemctl list-units –type=service –state=failed
• Confirm no post-install hooks will trigger power actions by simulating an upgrade on a test host:
  • apt update && apt upgrade –simulate

Testing backup restoration

• Restore a VM from backup to ensure backups are valid. Use vzdump restores or your backup tool.
• Test boot and network of critical appliances from the backup.

Monitoring for future upgrades

• Before any Proxmox upgrade, list installed packages and flag non-standard ones:
  • apt list –installed | grep -vE ‘proxmox|pve|pve-manager|pve-kernel’
• Maintain a pre-upgrade checklist:
  • Snapshot or backup configuration and VMs
  • Drain non-essential services from the hypervisor
  • Move hardware-control packages off the host
• Automate monitoring of apt logs and systemd failures post-upgrade:
  • journalctl -b -1 -p err

Concrete takeaways

• Treat the hypervisor as minimal. Keep only hypervisor-critical packages on it. That is hypervisor best practices.
• Check every package that touches hardware before a Proxmox upgrade.
• Keep backups and spare hardware. Use tested recovery steps: boot an older kernel, run dpkg –configure -a, repair or remove offending packages.
• Manage UPS on dedicated systems or isolated VMs, not on the hypervisor itself. That protects system stability and avoids a single package wrecking your network.