Motivation

Hey! A few months ago, my VMWare license expired. I decided to give Proxmox a try. Since I made a few changes in CLI, I thought I’d write them down to remember them.

Proxmox settings

Power consumption

My hardware is running 24/7. I wanted to reduce the power consumption as much as possible. I was able to reduce the power consumption from 34W to 12W during idle. I allready wrote a blog post about this topic. You can find it here.

# check which CPU governor modes are available
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

# check which CPU governor mode is currently in use
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# check the current CPU frequency
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq

# set the CPU governor mode to powersave
echo "powersave" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# alternatively, you can use the following command
cat /proc/cpuinfo | grep -E 'processor|MHz'

To permanently set the CPU governor mode to powersave, I used cron (not the best solution, but it works):

crontab -e

Add the following line:

# CPU-Powermode
@reboot echo "powersave" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null 2>&1

Swap on ZFS

While this is not recommended at all, there might be cases where you still want to have swap on ZFS. I did this by creating a ZVOL and using it as swap:

zfs create -V 128G -b $(getconf PAGESIZE) -o logbias=throughput -o sync=always -o primarycache=metadata -o com.sun:auto-snapshot=false rpool/swap
mkswap -f /dev/zvol/rpool/swap
swapon /dev/zvol/rpool/swap
echo '/dev/zvol/rpool/swap none swap discard 0 0' >> /etc/fstab

ZFS ARC tuning

Since I have 128GB of RAM in my server, I wanted to increase the ZFS ARC size to 32GB. This can be done by setting the zfs_arc_max parameter:

# set the ZFS ARC size minimum to 8GB
echo $[8 * 2 ** 30] >/sys/module/zfs/parameters/zfs_arc_min

# set the ZFS ARC size to 32GB
echo $[32 * 2 ** 30] >/sys/module/zfs/parameters/zfs_arc_max

# make the changes permanent
echo "options zfs zfs_arc_min=$[8 * 2 ** 30]" > /etc/modprobe.d/zfs.conf
echo "options zfs zfs_arc_max=$[32 * 2 ** 30]" >> /etc/modprobe.d/zfs.conf
update-initramfs -u

ZFS commands

# scrub the ZFS pool
zpool scrub rpool

# check the status of the ZFS pool
zpool status -v

Install sensors

apt-get install lm-sensors
sensors-detect
watch -n 1 sensors

Backup LXC containers to NFS target

TODO: check if this is still needed

Since I’m using the integrated backup solution, to backup my LXC containers to a NFS target, I had to make a few changes to make it work. Without those, the backup would fail with the following error:

INFO: temporary directory is on NFS, disabling xattr and acl support,
consider configuring a local tmpdir via /etc/vzdump.conf

I fixed this by adding the following to my /etc/vzdump.conf:

tmpdir: /tmp/

Change cluster votes

nano /etc/pve/corosync.conf

Temporary disable quorum

pvecm expected 1

Delete a node from the cluster

pvecm delnode <nodename>

Force remove all cluster information

rm -f /etc/pve/cluster.conf /etc/pve/corosync.conf
rm -f /etc/cluster/cluster.conf /etc/corosync/corosync.conf
rm /var/lib/pve-cluster/.pmxcfs.lockfile

GPU passthrough

AMD server

Add the following to your /etc/modules:

vfio
vfio_iommu_type1
vfio_pci

update-grub
reboot

I also had to enable CSM in the BIOS for the iGPU pass-through to work.

Intel server

Add the following to your /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
intel_iommu=on
iommu=pt

Add the following to your /etc/modules:

vfio
vfio_iommu_type1
vfio_pci

Add the following to your /etc/modprobe.d/blacklist.conf:

blacklist amdgpu
blacklist radeon
blacklist nouveau
blacklist nvidia

First, get the PCI IDs of your GPU:

lspci -nnk | grep -i vga -A3
lspci -n -s 01:00

Add the following to your /etc/modprobe.d/vfio.conf, replacing the PCI IDs with your own:

options vfio-pci ids=10de:2504,10de:228e

Finally update grub & reboot:

update-grub
reboot

After reboot, you can check if the IOMMU is enabled:

dmesg | grep -e DMAR -e IOMMU

should return

DMAR: IOMMU enabled

Bonus “tricks”

Using .ova images

Sometimes, my university provides a .ova image to work with. It’s quite easy to use it within Proxmox:

tar -xvf WindowsITSi.ova
qm importdisk 9090 WindowsITSi-disk001.vmdk ZFS_mirror --format qcow2