RoseWiki - User contributions [en]

Proxmox Remove Ceph old health warnings

2024-12-17T16:36:21Z

Julian: Created Page

If you have solved a problem with '''Proxmox''' and Ceph, e.g. the crash of a monitor or OSD deamon has been noted and corrected, the error message often remains in the Proxmox web GUI. These log entries in the GUI are crash logs of the underlying Ceph system. The Ceph crash command can be used to manage Ceph crash logs:

=== List Ceph crashes ===
root@pve-01:~# ceph crash ls

ID ENTITY NEW
2020-10-26_15:18:33.471228Z_1e218df6-0c92-4269-8e03-3bf6564e9aac mon.pve-03
2020-10-26_15:29:58.556281Z_1cda3f71-d4a1-47e2-83e9-a08285d2e041 mon.pve-02

=== Hide Ceph Crashes from the Proxmox GUI ===

* ceph crash archive <ID>: Archives single crash entry (will not appear in Proxmox GUI anymore)
* ceph crash archive-all: Archives all crash entries (no longer appear in the Proxmox GUI)

After archiving, the crashes are still viewable with ceph crash ls.

=== Ceph crash commands ===

* ceph crash info <ID >: Show details about the specific crash
* ceph crash stat: Shows the number of crashes since Ceph installation
* ceph crash rm <ID>: Deletes a single crash entry
* ceph crash prune <DAYS>: Deletes crashes older than <DAYS> days

Author: Jonas Sterr

Category:Linux Tutorials

2024-12-17T16:35:33Z

Julian: /* Linux Tutorials (Especially helpful for Proxmox) */

Here are some of our Linux tutorials.

== Linux Tutorials (Especially helpful for Proxmox) ==

* [[Offline Uncorrectable Sectors]]
* [[ZFS Failed Disk Replacement]]
* [[Calculating SSD Wearout]]
* [[Proxmox Backup Server Replication]]
* [[Proxmox Host SSH keys]]
* [[Replacing Proxmox Virtual Environment Server in a Ceph cluster]]
* [[Proxmox Remove Ceph old health warnings]]

Proxmox Host SSH keys

2024-12-04T00:06:39Z

Julian:

Follow these steps to regenerate OpenSSH Host Keys after cloning a Proxmox host.

Delete old ssh host keys: rm /etc/ssh/ssh_host_*
Reconfigure OpenSSH Server: dpkg-reconfigure openssh-server
Update all ssh client(s) ~/.ssh/known_hosts files

pvecm updatecerts -f

ufear said:
So, if anybody runs into this. I couldn't get updatecerts to add keys for reinstalled nodes to the global /etc/pve/priv/ssh_known_hosts; however the folder /etc/pve/nodes/<nodename> contains a ssh_known_hosts file which contains the content you need; copy it over and the world is good again.

Your post put me in the right track and it seems I'm able to connect by WebGUI shell from any host to any host in the cluster now.

The problem was that two of my nodes were missing ssh_known_hosts file in
Code:

/etc/pve/nodes/<node>/

(The hosts that gave me KEY CHANGED warning in WebGUI Shell)

I logged in to both troublesome nodes via ssh terminal and copied SSH public key from
Code:

/etc/ssh/ssh_host_rsa_key.pub

to
Code:

/etc/pve/nodes/<node>/ssh_known_hosts

file and added the node hostname in the beginning of the line before RSA public key like so:

Code:

NodeHostname ssh-rsa <the_rsa_pub_key>

after that I restarted SSH service systemctl restart sshd on both nodes (not sure if necessary)

This seems to have worked.

Proxmox Host SSH keys

2024-12-04T00:06:02Z

Julian:

Follow these steps to regenerate OpenSSH Host Keys

Delete old ssh host keys: rm /etc/ssh/ssh_host_*
Reconfigure OpenSSH Server: dpkg-reconfigure openssh-server
Update all ssh client(s) ~/.ssh/known_hosts files

pvecm updatecerts -f

ufear said:
So, if anybody runs into this. I couldn't get updatecerts to add keys for reinstalled nodes to the global /etc/pve/priv/ssh_known_hosts; however the folder /etc/pve/nodes/<nodename> contains a ssh_known_hosts file which contains the content you need; copy it over and the world is good again.

Your post put me in the right track and it seems I'm able to connect by WebGUI shell from any host to any host in the cluster now.

The problem was that two of my nodes were missing ssh_known_hosts file in
Code:

/etc/pve/nodes/<node>/

(The hosts that gave me KEY CHANGED warning in WebGUI Shell)

I logged in to both troublesome nodes via ssh terminal and copied SSH public key from
Code:

/etc/ssh/ssh_host_rsa_key.pub

to
Code:

/etc/pve/nodes/<node>/ssh_known_hosts

file and added the node hostname in the beginning of the line before RSA public key like so:

Code:

NodeHostname ssh-rsa <the_rsa_pub_key>

after that I restarted SSH service systemctl restart sshd on both nodes (not sure if necessary)

This seems to have worked.

Proxmox Host SSH keys

2024-12-03T23:52:20Z

Julian:

pvecm updatecerts -f

ufear said:
So, if anybody runs into this. I couldn't get updatecerts to add keys for reinstalled nodes to the global /etc/pve/priv/ssh_known_hosts; however the folder /etc/pve/nodes/<nodename> contains a ssh_known_hosts file which contains the content you need; copy it over and the world is good again.

Your post put me in the right track and it seems I'm able to connect by WebGUI shell from any host to any host in the cluster now.

The problem was that two of my nodes were missing ssh_known_hosts file in
Code:

/etc/pve/nodes/<node>/

(The hosts that gave me KEY CHANGED warning in WebGUI Shell)

I logged in to both troublesome nodes via ssh terminal and copied SSH public key from
Code:

/etc/ssh/ssh_host_rsa_key.pub

to
Code:

/etc/pve/nodes/<node>/ssh_known_hosts

file and added the node hostname in the beginning of the line before RSA public key like so:

Code:

NodeHostname ssh-rsa <the_rsa_pub_key>

after that I restarted SSH service systemctl restart sshd on both nodes (not sure if necessary)

This seems to have worked.

Proxmox Host SSH keys

2024-12-03T23:49:06Z

Julian: Created page with " May 3, 2024 Add bookmark #59 ufear said: So, if anybody runs into this. I couldn't get updatecerts to add keys for reinstalled nodes to the global /etc/pve/priv/ssh_known_hosts; however the folder /etc/pve/nodes/<nodename> contains a ssh_known_hosts file which contains the content you need; copy it over and the world is good again. Your post put me in the right track and it seems I'm able to connect by WebGUI shell from any host to any host in..."

May 3, 2024

Add bookmark
#59

ufear said:
So, if anybody runs into this. I couldn't get updatecerts to add keys for reinstalled nodes to the global /etc/pve/priv/ssh_known_hosts; however the folder /etc/pve/nodes/<nodename> contains a ssh_known_hosts file which contains the content you need; copy it over and the world is good again.

Your post put me in the right track and it seems I'm able to connect by WebGUI shell from any host to any host in the cluster now.

The problem was that two of my nodes were missing ssh_known_hosts file in
Code:

/etc/pve/nodes/<node>/

(The hosts that gave me KEY CHANGED warning in WebGUI Shell)

I logged in to both troublesome nodes via ssh terminal and copied SSH public key from
Code:

/etc/ssh/ssh_host_rsa_key.pub

to
Code:

/etc/pve/nodes/<node>/ssh_known_hosts

file and added the node hostname in the beginning of the line before RSA public key like so:

Code:

NodeHostname ssh-rsa <the_rsa_pub_key>

after that I restarted SSH service systemctl restart sshd on both nodes (not sure if necessary)

This seems to have worked.

Category:Linux Tutorials

2024-12-03T23:48:44Z

Julian:

Category:Linux Tutorials

2024-12-03T23:47:21Z

Julian: /* Linux Tutorials (Especially helpful for Proxmox) */

ZFS Failed Disk Replacement

2024-12-01T05:29:02Z

Julian: Fix UEFI boot with proxmox-boot-tool

==Copy partitions from good disk sda to blank disk sdb==
sgdisk /dev/sda -R /dev/sdb # sgdisk /dev/sda<From this disk> -R /dev/sdb<Replicate to this disk> sgdisk -G /dev/sdb # randomize the GUID on the new disk since it was copied from the other drive.

==Using Parted to verify the partition table of /dev/sdb==
(parted) select /dev/sdb Using /dev/sdb
 (parted) p Model: ATA WDC WD2000FYYZ-0 (scsi) Disk /dev/sdb: 2000398934016B Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags
1 1048576B 2097151B 1048576B Grub-Boot-Partition bios_grub
2 2097152B 136314879B 134217728B fat32 EFI-System-Partition boot, esp
3 136314880B 2000397885439B 2000261570560B zfs PVE-ZFS-Partition

(Ok partitions copied)

==Copy data from /dev/sda1 to /dev/sdb1==
dd if=/dev/sda1 of=/dev/sdb1 bs=512 #This is the bios boot partition
root@folkvang:~# dd if=/dev/sda1 of=/dev/sdb1 bs=512
2014+0 records in
2014+0 records out
1031168 bytes (1.0 MB) copied, 0.10164 s, 10.1 MB/s

==Replace the failed partition in the zpool==
Find the ID of the failed block device
root@folkvang:~# zpool status
pool: rpool
state: DEGRADED
status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: scrub repaired 0 in 0h25m with 0 errors on Sun May 8 11:20:27 2016
config
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
993077023721924477 FAULTED 0 0 0 was /dev/sdk2
sdk2 ONLINE 0 0 0
errors: No known data errors

==Call zpool to replace the failed device==
root@folkvang:~# zpool replace -f rpool 993077023721924477 /dev/sdl2

Make sure to wait until resilver is done before rebooting.
root@folkvang:~# zpool statuspool: rpool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Sep 2 16:45:53 2016 13.2M scanned out of 8.83G at 902K/s, 2h50m to go 12.9M resilvered, 0.15% done config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 replacing-0 UNAVAIL 0 0 0 993077023721924477 FAULTED 0 0 0 was /dev/sdk2 sdl2 ONLINE 0 0 0 (resilvering) sdk2 ONLINE 0 0 0 errors: No known data errors

== After fixing the drive, we need to ensure that the boot sectors are configured. ==
proxmox-boot-tool format /dev/sdb2

proxmox-boot-tool init /dev/sdb2

proxmox-boot-tool refresh

proxmox-boot-tool status

proxmox-boot-tool clean

grub-install /dev/sdk grub-install /dev/sdl update-grub

[[Category:Linux Tutorials]]
[[Category:Proxmox Tutorials]]

ZFS Failed Disk Replacement

2024-10-13T14:27:24Z

Julian:

==Copy partitions from good disk sda to blank disk sdb==
sgdisk /dev/sda -R /dev/sdb # sgdisk /dev/sda<From this disk> -R /dev/sdb<Replicate to this disk> sgdisk -G /dev/sdb # randomize the GUID on the new disk since it was copied from the other drive.

==Using Parted to verify the partition table of /dev/sdb==
(parted) select /dev/sdb Using /dev/sdb
 (parted) p Model: ATA WDC WD2000FYYZ-0 (scsi) Disk /dev/sdb: 2000398934016B Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags
1 1048576B 2097151B 1048576B Grub-Boot-Partition bios_grub
2 2097152B 136314879B 134217728B fat32 EFI-System-Partition boot, esp
3 136314880B 2000397885439B 2000261570560B zfs PVE-ZFS-Partition

(Ok partitions copied)

==Copy data from /dev/sda1 to /dev/sdb1==
dd if=/dev/sda1 of=/dev/sdb1 bs=512 #This is the bios boot partition
root@folkvang:~# dd if=/dev/sda1 of=/dev/sdb1 bs=512
2014+0 records in
2014+0 records out
1031168 bytes (1.0 MB) copied, 0.10164 s, 10.1 MB/s

==Replace the failed partition in the zpool==
Find the ID of the failed block device
root@folkvang:~# zpool status
pool: rpool
state: DEGRADED
status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: scrub repaired 0 in 0h25m with 0 errors on Sun May 8 11:20:27 2016
config
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
993077023721924477 FAULTED 0 0 0 was /dev/sdk2
sdk2 ONLINE 0 0 0
errors: No known data errors

==Call zpool to replace the failed device==
root@folkvang:~# zpool replace -f rpool 993077023721924477 /dev/sdl2

Make sure to wait until resilver is done before rebooting.
root@folkvang:~# zpool statuspool: rpool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Sep 2 16:45:53 2016 13.2M scanned out of 8.83G at 902K/s, 2h50m to go 12.9M resilvered, 0.15% done config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 replacing-0 UNAVAIL 0 0 0 993077023721924477 FAULTED 0 0 0 was /dev/sdk2 sdl2 ONLINE 0 0 0 (resilvering) sdk2 ONLINE 0 0 0 errors: No known data errors

== After fixing the drive, we need to ensure that the boot sectors are configured. ==
grub-install /dev/sdk grub-install /dev/sdl update-grub

[[Category:Linux Tutorials]]
[[Category:Proxmox Tutorials]]

ZFS Failed Disk Replacement

2024-10-09T14:33:06Z

Julian: /* Copy partitions from good disk sda to blank disk sdb */

==Copy partitions from good disk sda to blank disk sdb==
sgdisk /dev/sda -R /dev/sdb # sgdisk /dev/sda<From this disk> -R /dev/sdb<Replicate to this disk> sgdisk -G /dev/sdb # randomize the GUID on the new disk since it was copied from the other drive.

==Using Parted to verify the partition table of /dev/sdl==
(parted) select /dev/sdl Using /dev/sdl
 (parted) p Model: ATA WDC WD2000FYYZ-0 (scsi) Disk /dev/sdl: 2000398934016B Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags
1 1048576B 2097151B 1048576B Grub-Boot-Partition bios_grub
2 2097152B 136314879B 134217728B fat32 EFI-System-Partition boot, esp
3 136314880B 2000397885439B 2000261570560B zfs PVE-ZFS-Partition

(Ok partitions copied)

==Copy data from /dev/sda1 to /dev/sdb1 and /dev/sda2 to /dev/sdb2==
dd if=/dev/sda1 of=/dev/sdb1 bs=512 #This is the bios boot partition
root@folkvang:~# dd if=/dev/sdk1 of=/dev/sdl1 bs=512
2014+0 records in
2014+0 records out
1031168 bytes (1.0 MB) copied, 0.10164 s, 10.1 MB/s

==Replace the failed partition in the zpool==
Find the ID of the failed block device
root@folkvang:~# zpool status
pool: rpool
state: DEGRADED
status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: scrub repaired 0 in 0h25m with 0 errors on Sun May 8 11:20:27 2016
config
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
993077023721924477 FAULTED 0 0 0 was /dev/sdk2
sdk2 ONLINE 0 0 0
errors: No known data errors

==Call zpool to replace the failed device==
root@folkvang:~# zpool replace -f rpool 993077023721924477 /dev/sdl2

Make sure to wait until resilver is done before rebooting.
root@folkvang:~# zpool statuspool: rpool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Sep 2 16:45:53 2016 13.2M scanned out of 8.83G at 902K/s, 2h50m to go 12.9M resilvered, 0.15% done config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 replacing-0 UNAVAIL 0 0 0 993077023721924477 FAULTED 0 0 0 was /dev/sdk2 sdl2 ONLINE 0 0 0 (resilvering) sdk2 ONLINE 0 0 0 errors: No known data errors
After fixing the drive, we need to ensure that the boot sectors are configured.

grub-install /dev/sdk grub-install /dev/sdl update-grub

[[Category:Linux Tutorials]]
[[Category:Proxmox Tutorials]]