RoseWiki - Recent changes [en]

ZFS Failed Disk Replacement

Maeve — Wed, 17 Jun 2026 18:46:44 GMT

← Older revision		Revision as of 18:46, 17 June 2026
(One intermediate revision by the same user not shown)
Line 1:		Line 1:
			== Identify dead drive (optional) ==
			Assuming that a failed disk is still electronically active and functional, and that only its actual storage medium has failed, there is a way to identify in the chassis which disk is failed, should they be unlabeled.

			This can be done by using a utility called '''ledctl''', which is part of the '''ledmon''' package on apt-based distros such as Proxmox (and also Ubuntu, Debian).

			First, install ledmon
			apt install ledmon
			You'll notice while this is installing that it creates a service. After you're done using it you could disable this if desired with '''systemctl disable ledmon''' but for now we'll leave it running.

			ledctl has two options, both of which take the ID for a disk as an argument. These are locate and locate_off. For example, to make disk /dev/sdb blink continuously, you can use
			ledctl locate=/dev/sdb
			And then to stop the flashing, we simply run this with locate_off:
			ledctl locate_off=/dev/sdb

			=== Running ledctl as sudo ===
			The above assumes that we are using the root user. If not logged in as root, you must either su - to root or use sudo as this is a command explicitly referencing hardware, which can't be done by other users.

	==Copy partitions from good disk sda to blank disk sdb==		==Copy partitions from good disk sda to blank disk sdb==
	sgdisk /dev/sda -R /dev/sdb # sgdisk /dev/sda<From this disk> -R /dev/sdb<Replicate to this disk><br>sgdisk -G /dev/sdb # randomize the GUID on the new disk since it was copied from the other drive.		sgdisk /dev/sda -R /dev/sdb # sgdisk /dev/sda<From this disk> -R /dev/sdb<Replicate to this disk><br>sgdisk -G /dev/sdb # randomize the GUID on the new disk since it was copied from the other drive.

Proxmox ZFS Pool Setup

Maeve — Fri, 29 May 2026 19:08:06 GMT

← Older revision		Revision as of 19:08, 29 May 2026
(One intermediate revision by the same user not shown)
Line 2:		Line 2:

	== Prerequisites ==		== Prerequisites ==
	This process is being done to a node that has two drives, sized 4TB. We will be setting a 250GB partition on each drive to serve as the local/local-zfs storage in Proxmox. The remaining free space on the drives will be used to create a second Zpool. During the installation of Proxmox we set the hdsize parameter to 250GB. This will ensure it only uses 250GB of the boot drives for the configuration, leaving the rest untouched. After this, we need to use '''cfdisk''' to create partitions out of the remaining ~~storage data~~. We will leave this without a filesystem as ZFS will be installing its filesystem over these partitions in the next step.		This process is being done to a node that has two drives, sized 4TB. We will be setting a 250GB partition on each drive to serve as the local/local-zfs storage in Proxmox. The remaining free space on the drives will be used to create a second Zpool. During the installation of Proxmox we set the hdsize parameter to 250GB. This will ensure it only uses 250GB of the boot drives for the configuration, leaving the rest untouched. After this, we need to use '''cfdisk''' to create partitions out of the remaining disk space. We will leave this without a filesystem as ZFS will be installing its filesystem over these partitions in the next step.

	== Creating the main pool ==		== Creating the main pool ==
	''(The following has been adapted from the results of zpool history on an existing Zpool.)''		''(The following has been adapted from the results of zpool history on an existing Zpool.)''
	zpool create -o ashift=12 <pool name> mirror /dev/disk/by-id/disk1-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX-~~part3~~ /dev/disk/by-id/disk2-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX-~~part3~~		zpool create -o ashift=12 <pool name> mirror /dev/disk/by-id/disk1-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX-part4 /dev/disk/by-id/disk2-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX-part4
	zfs set compression=zstd <pool name>		zfs set compression=zstd <pool name>
	zfs create <pool name>/vmstorage		zfs create <pool name>/vmstorage
Line 15:		Line 15:
	zfs set compression=zstd <pool name>/vmstorage		zfs set compression=zstd <pool name>/vmstorage
	zfs set compression=zstd <pool name>/vz-dir		zfs set compression=zstd <pool name>/vz-dir
	In the above example, the suffix of -~~part3~~ is inferred for demonstrative purposes. Depending on the exact configuration set when installing Proxmox and creating its default local/local-zfs system as well as later using cfdisk, this could be a different partition number.		In the above example, the suffix of -part4 is inferred for demonstrative purposes. Depending on the exact configuration set when installing Proxmox and creating its default local/local-zfs system as well as later using cfdisk, this could be a different partition number.

Proxmox ZFS Pool Setup

Maeve — Fri, 29 May 2026 18:30:10 GMT

Created page with "The goal of this procedure is to set up a ZFS pool (Zpool) and its subpools on a Proxmox server that doesn't have dedicated disks for storage ZFS. == Prerequisites == This process is being done to a node that has two drives, sized 4TB. We will be setting a 250GB partition on each drive to serve as the local/local-zfs storage in Proxmox. The remaining free space on the drives will be used to create a second Zpool. During the installation of Proxmox we set the hdsize para..."

New page

The goal of this procedure is to set up a ZFS pool (Zpool) and its subpools on a Proxmox server that doesn't have dedicated disks for storage ZFS.

== Prerequisites ==
This process is being done to a node that has two drives, sized 4TB. We will be setting a 250GB partition on each drive to serve as the local/local-zfs storage in Proxmox. The remaining free space on the drives will be used to create a second Zpool. During the installation of Proxmox we set the hdsize parameter to 250GB. This will ensure it only uses 250GB of the boot drives for the configuration, leaving the rest untouched. After this, we need to use '''cfdisk''' to create partitions out of the remaining storage data. We will leave this without a filesystem as ZFS will be installing its filesystem over these partitions in the next step.

== Creating the main pool ==
''(The following has been adapted from the results of zpool history on an existing Zpool.)''
zpool create -o ashift=12 <pool name> mirror /dev/disk/by-id/disk1-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX-part3 /dev/disk/by-id/disk2-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX-part3
zfs set compression=zstd <pool name>
zfs create <pool name>/vmstorage
zfs create <pool name>/vz-dir
zfs set sync=disabled <pool name>/vmstorage
zfs set atime=on <pool name>
zfs set atime=off <pool name>/vmstorage
zfs set compression=zstd <pool name>/vmstorage
zfs set compression=zstd <pool name>/vz-dir
In the above example, the suffix of -part3 is inferred for demonstrative purposes. Depending on the exact configuration set when installing Proxmox and creating its default local/local-zfs system as well as later using cfdisk, this could be a different partition number.

Proxmox Host SSH keys

Maeve — Thu, 28 May 2026 21:21:08 GMT

← Older revision		Revision as of 21:21, 28 May 2026
Line 1:		Line 1:
	== ~~Intended method:~~ ==		== Introduction ==
	Delete old ssh host keys:		Proxmox nodes in a cluster communicate with each other through the Secure Shell (SSH) protocol. The SSH protocol is configured to require authentication by encryption keys. The host stores these keys in /root/.ssh/id_rsa.pub and in /etc/ssh/ssh_host_rsa_key.pub. When removing or replacing a Proxmox node, manual cleanup must be done. You must remove the old key from /etc/pve/priv/authorized_keys, and if joining a new machine in its place, add the new one.

			== Removing keys from removed machine. ==
			Delete old ssh host keys on the host you're troubleshooting or have removed with:
	rm /etc/ssh/ssh_host_*		rm /etc/ssh/ssh_host_*
	~~Reconfigure~~ OpenSSH ~~Server:~~		After doing this, we need to make sure the fingerprint of the old key is no longer on the other machines. Navigate to ''/etc/pve/priv/known_hosts'' and remove the fingerprint with the hostname of the machine you're treating. Also remove the key itself from /etc/pve/priv/authorized_keys.

			== Adding key ==
			Then we can recreate the keys with OpenSSH.
	dpkg-reconfigure openssh-server		dpkg-reconfigure openssh-server
	~~Update all~~ ssh ~~client(s) at ~~~/.~~ssh~~/~~known_hosts~~		After doing this we must copy the new key from /etc/ssh/ssh_host_rsa_key.pub on the treated machine to /etc/pve/priv/authorized keys. You will most likely have to do this manually copying from the broken machine to a machine still in the cluster, as with the keys removed, corosync may not copy the files correctly. After this we want to make sure that the fingerprint has been updated. On the treated machine, run:

	~~Then update certs and~~ keys ~~''on each~~ machine'':
	pvecm updatecerts -f		pvecm updatecerts -f

	== Manual ~~method~~<ref>https://forum.proxmox.com/threads/pvecm-updatecert-f-not-working.135812/page-3#post-660500</ref>: ==		== Manual correction if the above fails<ref>https://forum.proxmox.com/threads/pvecm-updatecert-f-not-working.135812/page-3#post-660500</ref> ==
	If this fails (which it might), ~~log into each troublesome node through SSHd and~~ copy the public key from		If this fails (which it might), copy the public key of the troublesome node from ''/etc/ssh/ssh_host_rsa_key.pub'' to ''/etc/pve/nodes/<node>/ssh_known_hosts'' and prepend it with that machine's hostname. Assuming a hostname of pve1, this line should appear as
	/etc/ssh/ssh_host_rsa_key.pub.
	~~Copy this~~ to
	/etc/pve/nodes/<node>/ssh_known_hosts
	and prepend it with that machine's hostname. Assuming a hostname of pve1, this line should appear as
	pve1 ssh-rsa <key>		pve1 ssh-rsa <key>
	Then restart the SSH daemon:		Then restart the SSH daemon:
	systemctl restart sshd		systemctl restart sshd