RoseWiki - User contributions [en]

Category:Linux Tutorials

2025-12-08T21:22:06Z

Maeve:

Here are some of our Linux tutorials.

== Linux Tutorials (Especially helpful for Proxmox) ==

* [[Offline Uncorrectable Sectors]]
* [[ZFS Failed Disk Replacement]]
* [[Calculating SSD Wearout]]
* [[Proxmox Backup Server Replication]]
* [[Proxmox Host SSH keys]]
* [[Proxmox Upgrade From 8 To 9]]
* [[Replacing A Proxmox Virtual Environment Server in a Ceph cluster]]
* [[Proxmox Backup Server Namespaces]]
* [[Proxmox Remove Ceph old health warnings]]

Proxmox Upgrade From 8 To 9

2025-12-08T21:19:22Z

Maeve: Created page with "== Initial Requirements == In order to upgrade from Proxmox 8 to 9, we need to ensure that a few minimal requirements are met to ensure that both the upgrade goes well and that guests do not experience incompatibilities afterwards. First, ensure that all members of the cluster are on the latest version of Proxmox 8 and Debian 12. This can be done by executing ''apt update'' and then ''apt dist-upgrade.'' If CEPH is installed, it must also be on the latest version of 19...."

== Initial Requirements ==
In order to upgrade from Proxmox 8 to 9, we need to ensure that a few minimal requirements are met to ensure that both the upgrade goes well and that guests do not experience incompatibilities afterwards.

First, ensure that all members of the cluster are on the latest version of Proxmox 8 and Debian 12. This can be done by executing ''apt update'' and then ''apt dist-upgrade.'' If CEPH is installed, it must also be on the latest version of 19.2. If CEPH is running 18, upgrade it beforehand. The upgrade will take at least 5GB storage on the root installation drive for each node, but it's preferable to have at least 10GB free.

Start by executing the bundled ''pve8to9'' command and checking its output. "Warnings" are details that may lead to incompatibilities or inconsistencies upon rebooting the host or that may disrupt service of hosts during the upgrade. The most important thing is to move running VMs off of the node being upgraded via either manual migration or HA, or to power them off if they're not critical.

=== QEMU version change ===
Due to Proxmox 9 sunsetting QEMU versions below 6, the first thing you should do is go to each VM's hardware options, check the Machine, and ensure that it's set to the latest version of QEMU, presently 10.1. After setting this, reboot each affected machine and ensure that the network interface and all drives are online. This is known to be a problem point.

=== 'screen' command ===
It's strongly, strongly advised that you run ''apt install screen'' before we continue. Screen is a terminal multiplexer that we will use to prevent potential failure due to dropped sessions if a network outage occurs.

=== pve8to9 checklist ===
Run pve8to9 from the command console. This will alert you to any potential breaking changes you need to watch for, such as quorum being insufficient for an upgrade or a change to the bootloader that must be accommodated before reboot after an otherwise successful upgrade. This is straightforward and you can broadly copy and paste the commands it tells you to execute.

== Update repositories ==

=== Debian and Proxmox repositories ===
Debian 13 uses a new package format called deb822. We start by adding the Proxmox repository as a deb822.
cat > /etc/apt/sources.list.d/proxmox.sources << EOF
Types: deb
URIs: <nowiki>http://download.proxmox.com/debian/pve</nowiki>
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
If using an enterprise repository, use the following:
cat > /etc/apt/sources.list.d/proxmox-enterprise.sources << EOF
Types: deb
URIs: <nowiki>https://enterprise.proxmox.com/debian/pve</nowiki>
Suites: trixie
Components: pve-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
Then let's ensure that we have the Debian repositories updated to 13, codenamed Trixie:
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
Manually check using nano or vim the various files in /etc/apt/sources.list. Any third party repositories such as zabbix should either be disabled by prefacing their lines with # or be updated to the trixie equivalent if possible. If a Debian 13 package exists for the third-party repository, simply change ''bookworm'' to ''trixie''. Also, at this stage, ''remove any references to Proxmox outside of our new proxmox.sources file''.

=== CEPH repositories ===
For CEPH repositories we follow the same procedure. Remove any references to CEPH in existing files in /etc/apt/sources.list and sources.list.d. Then add the deb822:
cat > /etc/apt/sources.list.d/ceph.sources << EOF
Types: deb
URIs: <nowiki>http://download.proxmox.com/debian/ceph-squid</nowiki>
Suites: trixie
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
For enterprise, use the following:
cat > /etc/apt/sources.list.d/ceph.sources << EOF
Types: deb
URIs: <nowiki>https://enterprise.proxmox.com/debian/ceph-squid</nowiki>
Suites: trixie
Components: enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
Update the repositories by running:
apt update
apt policy
After running apt policy, a page will appear showing the various sources for enabled repositories. Ensure that none say "bookworm".

== Upgrade ==
'''NEVER''' perform a Proxmox upgrade from the web UI console. Connect via SSH. Start a screen session:
screen -S proxmoxupgrade
At this point we can start the update.
# NEVER use apt upgrade. apt full-upgrade and apt dist-upgrade are synonyms, but all official Proxmox guides suggest dist-upgrade.
apt dist-upgrade
At various points during the upgrade, you will be prompted to replace existing config files with new ones from package maintainers. Here are some guidelines on which ones to allow to be replaced and which ones to keep:

'''/etc/issue''' - N (do not replace)

'''/etc/lvm/lvm.conf''' - Y (replace)

'''/etc/ssh/sshd_config''' - N

'''/etc/default/grub''' - You will only be asked this if there's been a meaningful change to the grub configuration. It's recommended you view the difference between the two. If the only difference is the addition of some bash scripting, accept the new file. If any lines differ in boot execution, consider carefully.

'''/etc/chrony/chrony.conf''' - Only prompted if custom configurations have been made. /etc/chrony/chrony.conf.d has been deprecated and any config lines in files under this directory must be moved to /etc/chrony/chrony.conf. It's unlikely that this has occurred and you will likely not be prompted.

== After Upgrade ==
Always reboot a node after an upgrade to ensure that it's running the latest Linux kernel and all packages have been reloaded.

Be aware that Proxmox 9 deprecates "HA groups" in favor of "HA rules" - this is an automatic conversion - this is triggered after all cluster members are on 9 or above.

You may now run ''apt modernize-sources'' which will convert any old repository configurations to the deb822 format.

It's advised to run ''systemctl disable --now systemd-journald-audit.socket'' to prevent overly long kernel audit messages during the upgrade

If booting from lvm in UEFI mode we have to run ''[ -d /sys/firmware/efi ] && apt install grub-efi-amd64''

Some systems' lvm thin pools may report an error, run lvconvert --repair pve/data to resolve.

ZFS Failed Disk Replacement

2025-07-12T21:27:11Z

Maeve:

==Copy partitions from good disk sda to blank disk sdb==
sgdisk /dev/sda -R /dev/sdb # sgdisk /dev/sda<From this disk> -R /dev/sdb<Replicate to this disk> sgdisk -G /dev/sdb # randomize the GUID on the new disk since it was copied from the other drive.

==Using Parted to verify the partition table of /dev/sdb==
(parted) select /dev/sdb Using /dev/sdb
 (parted) p Model: ATA WDC WD2000FYYZ-0 (scsi) Disk /dev/sdb: 2000398934016B Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags
1 1048576B 2097151B 1048576B Grub-Boot-Partition bios_grub
2 2097152B 136314879B 134217728B fat32 EFI-System-Partition boot, esp
3 136314880B 2000397885439B 2000261570560B zfs PVE-ZFS-Partition

(Ok partitions copied)

==Copy data from /dev/sda1 to /dev/sdb1==
dd if=/dev/sda1 of=/dev/sdb1 bs=512 #This is the bios boot partition
root@folkvang:~# dd if=/dev/sda1 of=/dev/sdb1 bs=512
2014+0 records in
2014+0 records out
1031168 bytes (1.0 MB) copied, 0.10164 s, 10.1 MB/s

==Replace the failed partition in the zpool==
Find the ID of the failed block device
root@folkvang:~# zpool status
pool: rpool
state: DEGRADED
status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: scrub repaired 0 in 0h25m with 0 errors on Sun May 8 11:20:27 2016
config
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
993077023721924477 FAULTED 0 0 0 was /dev/sdk2
sdk2 ONLINE 0 0 0
errors: No known data errors

==Call zpool to replace the failed device==
root@folkvang:~# zpool replace -f rpool 993077023721924477 /dev/sdl2

Make sure to wait until resilver is done before rebooting.
root@folkvang:~# zpool statuspool: rpool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Sep 2 16:45:53 2016 13.2M scanned out of 8.83G at 902K/s, 2h50m to go 12.9M resilvered, 0.15% done config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 replacing-0 UNAVAIL 0 0 0 993077023721924477 FAULTED 0 0 0 was /dev/sdk2 sdl2 ONLINE 0 0 0 (resilvering) sdk2 ONLINE 0 0 0 errors: No known data errors

== After fixing the drive, we need to ensure that the boot sectors are configured. ==
proxmox-boot-tool format /dev/sdb2

proxmox-boot-tool init /dev/sdb2

proxmox-boot-tool refresh

proxmox-boot-tool status

proxmox-boot-tool clean

grub-install /dev/sdk

grub-install /dev/sdl

update-grub

== OPTIONAL: Set Zpool to Expand ==
We can set a Zpool to expand itself to fill all available storage by first taking the pool offline with '''zpool offline <zpool>.''' Now, we can bring it back online with a special flag that will tell it to expand to fill its storage space. Sometimes this is enabled by default, but if it appears not to be working, you can do this to rule this setting not being set out. We do this with '''zpool online -e <zpool>'''.

If you cloned these drives directly from one drive to a larger drive, this won't work, as the partition table hasn't been reconfigured. You'll have to move the end partition and then expand the first one before running '''zpool''' '''online -e'''.

[[Category:Linux Tutorials]]
[[Category:Proxmox Tutorials]]

Force Wireguard DNS Zone Override

2025-07-07T14:54:33Z

Maeve:

Sometimes you may have a situation in which you require a machine using a Wireguard tunnel to always use a specific DNS server to resolve specific zones. One case where we needed this was an Active Directory DNS zone that also matched a publicly available DNS zone. The publicly available records resolved to one end of a NAT'd IP range, but the AD DNS zone had records pointing to the other side. When connecting locally in the network, this was fine, but some Windows clients were resolving through the LAN's DNS regardless of the search domain set in the Wireguard config.

To fix this we can leverage the power of the [https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn593632(v=ws.11) Name Resolution Policy Table].

You could simply fix this by executing this command:
Add-DnsClientNrptRule -Comment 'NameOfVPNTunnel' -Namespace '.dnszone.com' -NameServers 203.0.113.0
This works, but the problem is that this rule will always be active. If you know for sure that the device's network situation will never change, this is fine, but if it's a mobile workstation that may be repurposed you should avoid this if possible. We can tell Wireguard to dynamically create and remove these rules using the PostUp and PostDown hooks in the Wireguard config. The problem is that these are disabled by default due to the potential room for abuse. The solution is to enable these using a registry edit:
reg add HKLM\Software\WireGuard /v DangerousScriptExecution /t REG_DWORD /d 1 /f

If this results in "The operation was completed successfully" we can now add these lines to our wireguard config:
PostUp = powershell.exe -Command "& { Add-DnsClientNrptRule -Comment 'NameOfVPNTunnel' -Namespace '.dnszone.com' -NameServers 203.0.113.0 }"
PostDown = powershell.exe -Command "& { Get-DnsClientNrptRule | where Comment -eq 'NameOfVPNTunnel' | foreach { Remove-DnsClientNrptRule -Name $_.Name -Force } }"

Force Wireguard DNS Zone Override

2025-07-03T17:41:01Z

Maeve: Created page with "Sometimes you may have a situation in which you require a machine using a Wireguard tunnel to always use a specific DNS server to resolve specific zones. One case where we needed this was an Active Directory DNS zone that also matched a publicly available DNS zone. The publicly available records resolved to one end of a NAT'd IP range, but the AD DNS zone had records pointing to the other side. When connecting locally in the network, this was fine, but some Windows clien..."

Sometimes you may have a situation in which you require a machine using a Wireguard tunnel to always use a specific DNS server to resolve specific zones. One case where we needed this was an Active Directory DNS zone that also matched a publicly available DNS zone. The publicly available records resolved to one end of a NAT'd IP range, but the AD DNS zone had records pointing to the other side. When connecting locally in the network, this was fine, but some Windows clients were resolving through the LAN's DNS regardless of the search domain set in the Wireguard config.

To fix this we can leverage the power of the [https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn593632(v=ws.11) Name Resolution Policy Table].

You could simply fix this by executing this command:
Add-DnsClientNrptRule -Comment 'NameOfVPNTunnel' -Namespace '.dnszone.com' -NameServers 203.0.113.0
This works, but the problem is that this rule will always be active. If you know for sure that the device's network situation will never change, this is fine, but if it's a mobile workstation that may be repurposed you should avoid this if possible. We can tell Wireguard to dynamically create and remove these rules using the PostUp and PostDown hooks in the Wireguard config. The problem is that these are disabled by default due to the potential room for abuse. The solution is to enable these using a registry edit:
Set-ItemProperty -Path HKLM:\SOFTWARE\Wireguard -Name DangerousScriptExecution -Type DWord -Value 1
If this results in "The operation was completed successfully" we can now add these lines to our wireguard config:
PostUp = powershell.exe -Command "& { Add-DnsClientNrptRule -Comment 'NameOfVPNTunnel' -Namespace '.dnszone.com' -NameServers 203.0.113.0 }"
PostDown = powershell.exe -Command "& { Get-DnsClientNrptRule | where Comment -eq 'NameOfVPNTunnel' | foreach { Remove-DnsClientNrptRule -Name $_.Name -Force } }"

Watchguard IKEv2 Certificates

2025-05-26T20:24:10Z

Maeve: Created page with "Watchguard Fireboxes have several VPN protocols available for their Mobile VPN service. When using IKEv2, there's a problem that can arise if using the built-in self-signed certificates which are generated automatically. The problem is that the Firebox uses an internal CA certificate to sign the certificate. When configuring the client machine, you install a certificate bundle from the Firebox. For pretty reasonable security reasons, however, you cannot move this interna..."

Watchguard Fireboxes have several VPN protocols available for their Mobile VPN service. When using IKEv2, there's a problem that can arise if using the built-in self-signed certificates which are generated automatically. The problem is that the Firebox uses an internal CA certificate to sign the certificate. When configuring the client machine, you install a certificate bundle from the Firebox. For pretty reasonable security reasons, however, you cannot move this internal CA certificate from one Firebox to another, meaning that when you upgrade the machine, you ''must'' redo all VPN configurations, as the root CA installed on each client machine no longer matches. This is hugely inconvenient. The solution is to use a certificate signed by a third party certificate. You can either use a major third party certificate like Sectigo, or you can host this in-house.

You ''can'' use Windows CA Server but this requires another license. The other solution is to use an open-source CA system like [https://smallstep.com/docs/step-ca/ step-ca].

== Step CA Configuration ==

=== Step CA Host Certificate Length ===
Step CA was originally designed for automated recertification of internal web services. For this reason the default length of a certificate is 24 hours. If you want your certificates to last longer when signing them, we first need to edit the configuration of Step.
# -- snippet start - /root/.step/configs/ca.json --
"authority": {
"claims": {
"maxTLSCertDuration":"175200h" # ~20 years
},
"provisioners": [ # -- snippet end --

== Firebox Certificates ==
The IKEv2 certificate authenticates based on two things: '''trust''' and '''SAN matching'''.

=== Trust ===
The benefit of using a third party certificate server is that the only thing you have to install on the client machines ''is the root CA'', not the certificate on the Firebox. This means that as long as the same certificate is used to sign any certificate assigned to the firebox, it will maintain trust and be authenticated. This means you can swap out the physical hardware an indefinite amount of times, change the hostname / IP of the machine, and as long as the rest of the certificate matches and is signed by the same root CA, no action (except for updating the endpoint address) has to be taken on the clients.

=== SAN matching ===
As long as certificate trust is maintained, the next thing you have to consider is the SANs ('''Subject Alternative Name''') on the certificate on the Firebox. The SAN is set as such in OpenSSL syntax:
subjectAltName=DNS:hostname.domainname.com,IP:203.0.113.2

Rustdesk

2025-04-28T20:50:07Z

Maeve: Created page with "'''Rustdesk''' is a Remote Access Software used by Rosemark Networks to administer support to our customers regardless of VPN connectivity status. == Installation and configuration == # [https://github.com/rustdesk/rustdesk/releases Download the latest version of Rustdesk from their official Github repository page. Download the 64-bit version MSI installer.] # When prompted during the install, uncheck ''Install'' ''Rustdesk Printer.'' # After finishing the installati..."

'''Rustdesk''' is a Remote Access Software used by Rosemark Networks to administer support to our customers regardless of VPN connectivity status.

== Installation and configuration ==

# [https://github.com/rustdesk/rustdesk/releases Download the latest version of Rustdesk from their official Github repository page. Download the 64-bit version MSI installer.]
# When prompted during the install, uncheck ''Install'' ''Rustdesk Printer.''
# After finishing the installation, from the Rustdesk main menu, select the hamburger menu (three lines near the window titlebar controls)
# Select ''Network'' settings.
# Press ''Unlock'' and select ''Relay/ID Server Settings''.
# Enter '''rd.rosemarknetworks.com''' as both the ID and Relay server.
# API Server remains blank.
# The key can be found on this secret page: [[RustdeskPublicKey|Rustdesk public key.]] You cannot access this page unless you a Sysop user on this Wiki.
# Press OK.
# Go to ''Security'' settings and select Permanent Password. This will prompt you to set your permanent password.
# Restart Rustdesk app.
# Make note of Rustdesk ID.

Secret:RustdeskPublicKey

2025-04-28T20:40:53Z

Maeve: Maeve moved page Secret:RustdeskPublicKey to RustdeskPublicKey

#REDIRECT [[RustdeskPublicKey]]

Proxmox Backup Server

2025-03-18T20:39:43Z

Maeve: Created page with "== PAGE IS WIP == This page is a work in progress as there is much to discuss and document regarding Proxmox Backup Server (PBS) and integrations with Proxmox Virtual Environment (PVE). == Overview == == Installation == == Datastore Creation == == Scheduled Jobs == === Prune Jobs === === Garbage Collection Jobs === === Verify Jobs === == Administration == === Notifications === ==== Datastore Notifications Caveat ==== When setting up the notification system in P..."

== PAGE IS WIP ==
This page is a work in progress as there is much to discuss and document regarding Proxmox Backup Server (PBS) and integrations with Proxmox Virtual Environment (PVE).

== Overview ==

== Installation ==

== Datastore Creation ==

== Scheduled Jobs ==

=== Prune Jobs ===

=== Garbage Collection Jobs ===

=== Verify Jobs ===

== Administration ==

=== Notifications ===

==== Datastore Notifications Caveat ====
When setting up the notification system in Proxmox Backup Server, ensure that you select Notification System in the Notification Mode options on each datastore you want to get notifications for.

Netutils Manual

2025-01-03T17:49:19Z

Maeve: Created page with "WIP Test"

WIP Test

Category:Software Tutorials

2025-01-03T17:49:06Z

Maeve:

Here are some of our tutorials and notes on software, both selfhosted FOSS and 3rd party products.

== Software Tutorials ==

* [[Firebox Content Inspection|Firebox Content Inspection (HTTPS Content Inspection)]]
* [[Netutils Manual]]
* [[Nextcloud]]
* [[Packetfence]]
* [[Proxy Server]] (high level)
** [[Reverse Proxy]] (use-case specific)
* [[Zabbix]]

Proxmox Backup Server Namespaces

2024-12-28T22:13:17Z

Maeve: Created page with "Proxmox Backup Server has a feature called '''namespaces'''. Namespaces allow you to organize parts of your datastore under different directories, and then target these directories individually under prune jobs and sync jobs. One usecase may be if you have a set of backups that you only want to keep the last 30 days of, but you have another that you wish to keep a copy of per day. Namespaces can be nested, with one inside another one, following directory slash notation..."

Proxmox Backup Server has a feature called '''namespaces'''. Namespaces allow you to organize parts of your datastore under different directories, and then target these directories individually under prune jobs and sync jobs. One usecase may be if you have a set of backups that you only want to keep the last 30 days of, but you have another that you wish to keep a copy of per day.

Namespaces can be nested, with one inside another one, following directory slash notation (/root/nested_namespace1/nested_namespace2). Internally, Proxmox Backup Server's datastore uses a "ns" directory, alongside a "vm" and "ct" directory to organize this content. So internally, a root namespace with three VMs and a nested namespace with two more VMs may appear as such:
root/
| vm/
| 100/
| 101/
| 102/
| ct/
| ns/
| nested/
| vm/
| 103/
| 104/
| ct/
Namespaces can be nested many times over, if desired. Another important usecase for namespaces is that they allow you to have VMs from several PVE clusters that have overlapping VMIDs.

== Sync Jobs ==
'''Note: this assumes sync jobs using pulls instead of pushes, which was and continues to be the standard paradigm intended for Proxmox Backup Server.''' In a recent update, the options for a push were added, but this is a bit limited. If using push jobs, adapt these notes with caution.

When configuring a sync job, the local and remote namespace may be specified, to avoid copying the full contents of the backup server to another backup server. You may use this to configure several identical backup servers in a mesh, with local machines pushing backups to the backup server, followed by remotes syncing from other sites' local servers.

* Site A may have a backup server with namespaces A and B.
* Site B may have the same configuration, with namespaces A and B.
* Site A backs up to its local backup server's A namespace.
* Later in the day, Site B's backup server syncs from Site A's A Namespace into its own A namespace, only populated by contents gathered from the sync job, while Site B backs up its own contents to its local backup server's B namespace.
* The same process occurs with Site A syncing Site B's namespace into Site A's B namespace.

Now, both sites have both copies of themselves, formed locally, and copies of each other. This can scale up with an arbitrary number of backup servers.

=== Nested namespaces ===
Sync jobs support nested namespaces. By default, when configuring a sync job, you specify source and destination namespaces. These can be as specific as you like. Note the "'''Max Depth'''" option, which defaults to "Full". This option controls how many namespaces deep the sync job will crawl. Without a value set, it will sync the entire directory all the way down its tree. If you set this to 0, it will sync any contents in the source namespace, but will not traverse past this specific namespace. With the above example, a Max Depth of 0 syncing on the root namespace would copy VMs 100, 101, and 102, but NOT the nested namespace or its content.

==== Nested namespaces will be created if not already present ====
In the above example, if we have another Proxmox Backup Server that has its own root namespace, but no nested subnamespace, and we sync the contents from root with Max Depth full to this second server's root, it will automatically generate the "nested" namespace and use it in any subsequent syncs. This is why it's important to understand your namespace structure, as a misalignment could lead to many erroneous namespaces being generated!

== Manipulating namespaces ==
{|
![[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
!DO NOT do ANY of this if a prune job, garbage collection, verify job, sync job, or standard backup is in progress. It'd be best to set this datastore to maintenance mode first!
|}

==== Moving VMs and CTs from one namespace to another ====
This is not implicitly supported by Proxmox Backup Server, but due to the fact that the underlying infrastructure is a very simple POSIX directory system, it is not difficult to manually perform this action.

# If a namespace has been created, but does not have any VMs in it yet, the vm directory will be absent, so we must create this manually.
## From the parent namespace directory: ''mkdir ns/<namespace>/vm''
## Set the owner to backup:backup: ''chown backup:backup ns/<namespace>/vm''
# Move the contents you wish to move:
## mv vm/<vm_id> ns/<namespace>/vm/<vm_id>

Category:Linux Tutorials

2024-12-28T21:25:02Z

Maeve: /* Linux Tutorials (Especially helpful for Proxmox) */

Here are some of our Linux tutorials.

== Linux Tutorials (Especially helpful for Proxmox) ==

* [[Offline Uncorrectable Sectors]]
* [[ZFS Failed Disk Replacement]]
* [[Calculating SSD Wearout]]
* [[Proxmox Backup Server Replication]]
* [[Proxmox Host SSH keys]]
* [[Replacing A Proxmox Virtual Environment Server in a Ceph cluster]]
* [[Proxmox Backup Server Namespaces]]
* [[Proxmox Remove Ceph old health warnings]]

Replacing A Proxmox Virtual Environment Server in a Ceph cluster

2024-12-24T19:45:30Z

Maeve:

== The Situation ==
Imagine a situation in which you have a cluster of Proxmox VE nodes as part of a hpyerconverged Ceph installation. You want to replace one of your nodes with a fully brand new installation of Proxmox VE. You don't intend to migrate its drives, you want to replace the node outright with new drives and a new installation of Proxmox VE. In order for this to work, there are a handful of things you need to consider to prevent 1. unnecessary Ceph rebalances 2. broken HA in Proxmox VE. The broken HA issue comes from the fact that Proxmox VE uses SSH internally to move data around and to run commands on remote nodes. These keys get messed up because Proxmox does not remove the keys when decommissioning a node.

== The Strategy ==
Migrate all virtual machines from the node. Ensure that you have made copies of any local data or backups that you want to keep. In addition, make sure to remove any scheduled replication jobs to the node to be removed.
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|Failure to remove replication jobs to a node before removing said node will result in the replication job becoming irremovable. Especially note that replication automatically switches direction if a replicated VM is migrated, so by migrating a replicated VM from a node to be deleted, replication jobs will be set up to that node automatically.
|}
Replacing a Proxmox Virtual Environment Server in hyperconverged Ceph configuration.

This guide assumes a four node cluster with hostnames node1-node4. We're assuming a replacement of node2, and for convenience, we're migrating to node1, but it could be to any other node in the cluster. In this guide, we are ''specifically'' replacing node2 with a new installation of PVE on an upgraded server that shares the same IP(s) and hostname, not migrating an existing installation to a new chassis.

=== Confirm cluster status (prerequisite, optional). ===
From node2's CLI, run ''pvecm nodes.''
node2# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node1
2 1 node2 (local)
3 1 node3
4 1 node4
Confirm that the node you intend to decommission is listed has the ''"(local)"'' identifier next to its name. This confirms we're on the right machine.

=== Migrate VM, CT, templates, storage off of node2. ===
Using HA, migrate running VM's from node2 to node1.

Manually migrate all remaining nonrunning VMs, CTs, and templates off of node2 to node1.

=== Decommission node from CEPH. ===

# Set OSDs on node2 to "out" and wait for rebalance.
#* [[File:Ceph osd management.png]]
#* On Node2, go to Ceph -> OSD -> Node2 -> per each OSD listed, select the OSD and press "Out" in the top right of the control bar.
# Following rebalance, stop and destroy all OSDs on node2.
#* In the same screen of the PVE gui, select each out OSD and press stop, and once stopped, under the "more" submenu, select "destroy".
# Remove the Ceph servers (monitor, manager, and metadata) from node2.
#* In Ceph -> Monitor, select node2's monitor and press stop, and then once stopped, press destroy.
#** [[File:Ceph mon.png]]
#* In Ceph -> Monitor, select node2's manager in the bottom panel, press stop, and once stopped, press destroy.
#** [[File:Ceph manager.png]]
#* In Ceph -> CephFS - Metadata Servers, stop and destroy the Metadata Server for node2.
#** [[File:Ceph mds.png]]
# On node2's CLI, clean up the Ceph CRUSH map and remove the host bucket using "ceph osd crush remove node2".
# From node1's CLI, run "pvecm delnode node2"

{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|''Important: Power off node2 before running pvecm delnode''. This is because the SSH keys and some internal references to the cluster may still exist on the node being removed, and it may attempt to perform sync operations within corosync and any distributed storage (i.e. Replicated ZFS pools).
|}
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|At this point, it is possible that you will receive an error message stating <code>Could not kill node (error = CS_ERR_NOT_EXIST)</code>. This does not signify an actual failure in the deletion of the node, but rather a failure in corosync trying to kill an offline node. Thus, it can be safely ignored.
|}

=== Confirm node deleted. ===
From node1's CLI, run ''pvecm nodes.''
node2# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node1 (local)
2 1 node3
3 1 node4

=== Clean up SSH Keys. ===
There is a '''major missing detail''' in the official documentation which is that if you intend to join a node into a cluster with the '''same hostname and IP''' the previous node had, everything will break if you don't take some prerequisite actions.

Power on the new node2 preconfigured with the same hostname and IP address.

From node1, SSH into the new node2. This will generate an error from the SSH client which will contain a line containing a command that will remove the key from the known hosts file. Use this command syntax to clear these keys:
ssh-keygen -f ''<nowiki/>'/etc/ssh/ssh_known_hosts''' -R 'XXX.XXX.XXX.XXX'
ssh-keygen -f ''<nowiki/>'/etc/ssh/ssh_known_hosts''' -R 'node2.domain.com'
Now attempt to SSH into node2 again. You will get a nearly identical error, only this time the path to the known_hosts file will be in the root user's .ssh directory as so:
ssh-keygen -f ''<nowiki/>'/root/.ssh/ssh_known_hosts''' -R 'XXX.XXX.XXX.XXX'
ssh-keygen -f ''<nowiki/>'/root/.ssh/ssh_known_hosts''' -R 'node2.domain.com'
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|This second series of commands, pointing to /root/, are ''not replicated'' across the cluster members, and must be done on all members of the cluster manually.
|}
After executing these commands, when we join the node to the cluster, SSH will work correctly.

=== Join node to cluster. ===

# Power on server and configure LOM such as Dell DRAC to use the correct IP address.
# Edit /etc/hostname and /etc/hosts to confirm hostname is correctly matched to previous install's hostname.
# Reboot and verify hostname and IP are correct.
# If the previous machine had a Proxmox license, apply it now.
# Validate network connectivty on corosync network and both the Ceph frontend (consumption and management) and backend (replication) networks to all other nodes.
# Join the Proxmox cluster.
# Install Ceph.
# Add Ceph monitor and Ceph manager to this node.
# Migrate a test VM to the new node to confirm consumption.
# If there are any other maintenance tasks to complete (like swapping another node with the previous node's hardware) do NOT add OSDs back to node2 until ready.

Replacing A Proxmox Virtual Environment Server in a Ceph cluster

2024-12-24T19:44:29Z

Maeve:

== The Situation ==
Imagine a situation in which you have a cluster of Proxmox VE nodes as part of a hpyerconverged Ceph installation. You want to replace one of your nodes with a fully brand new installation of Proxmox VE. You don't intend to migrate its drives, you want to replace the node outright with new drives and a new installation of Proxmox VE. In order for this to work, there are a handful of things you need to consider to prevent 1. unnecessary Ceph rebalances 2. broken HA in Proxmox VE. The broken HA issue comes from the fact that Proxmox VE uses SSH internally to move data around and to run commands on remote nodes. These keys get messed up because Proxmox does not remove the keys when decommissioning a node.

== The Strategy ==
Migrate all virtual machines from the node. Ensure that you have made copies of any local data or backups that you want to keep. In addition, make sure to remove any scheduled replication jobs to the node to be removed.
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|Failure to remove replication jobs to a node before removing said node will result in the replication job becoming irremovable. Especially note that replication automatically switches direction if a replicated VM is migrated, so by migrating a replicated VM from a node to be deleted, replication jobs will be set up to that node automatically.
|}
Replacing a Proxmox Virtual Environment Server in hyperconverged Ceph configuration.

This guide assumes a four node cluster with hostnames node1-node4. We're assuming a replacement of node2, and for convenience, we're migrating to node1, but it could be to any other node in the cluster. In this guide, we are ''specifically'' replacing node2 with a new installation of PVE on an upgraded server that shares the same IP(s) and hostname, not migrating an existing installation to a new chassis.

=== Confirm cluster status (prerequisite, optional). ===
From node2's CLI, run ''pvecm nodes.''
node2# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node1
2 1 node2 (local)
3 1 node3
4 1 node4
Confirm that the node you intend to decommission is listed has the ''"(local)"'' identifier next to its name. This confirms we're on the right machine.

=== Migrate VM, CT, templates, storage off of node2. ===
Using HA, migrate running VM's from node2 to node1.

Manually migrate all remaining nonrunning VMs, CTs, and templates off of node2 to node1.

=== Decommission node from CEPH. ===

# Set OSDs on node2 to "out" and wait for rebalance. [[File:Ceph osd management.png]]
#* On Node2, go to Ceph -> OSD -> Node2 -> per each OSD listed, select the OSD and press "Out" in the top right of the control bar.
# Following rebalance, stop and destroy all OSDs on node2.
#* In the same screen of the PVE gui, select each out OSD and press stop, and once stopped, under the "more" submenu, select "destroy".
# Remove the Ceph servers (monitor, manager, and metadata) from node2.
#* In Ceph -> Monitor, select node2's monitor and press stop, and then once stopped, press destroy. [[File:Ceph mon.png]]
#* In Ceph -> Monitor, select node2's manager in the bottom panel, press stop, and once stopped, press destroy. [[File:Ceph manager.png]]
#* In Ceph -> CephFS - Metadata Servers, stop and destroy the Metadata Server for node2. [[File:Ceph mds.png]]
# On node2's CLI, clean up the Ceph CRUSH map and remove the host bucket using "ceph osd crush remove node2".
# From node1's CLI, run "pvecm delnode node2"

{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|''Important: Power off node2 before running pvecm delnode''. This is because the SSH keys and some internal references to the cluster may still exist on the node being removed, and it may attempt to perform sync operations within corosync and any distributed storage (i.e. Replicated ZFS pools).
|}
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|At this point, it is possible that you will receive an error message stating <code>Could not kill node (error = CS_ERR_NOT_EXIST)</code>. This does not signify an actual failure in the deletion of the node, but rather a failure in corosync trying to kill an offline node. Thus, it can be safely ignored.
|}

=== Confirm node deleted. ===
From node1's CLI, run ''pvecm nodes.''
node2# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node1 (local)
2 1 node3
3 1 node4

=== Clean up SSH Keys. ===
There is a '''major missing detail''' in the official documentation which is that if you intend to join a node into a cluster with the '''same hostname and IP''' the previous node had, everything will break if you don't take some prerequisite actions.

Power on the new node2 preconfigured with the same hostname and IP address.

From node1, SSH into the new node2. This will generate an error from the SSH client which will contain a line containing a command that will remove the key from the known hosts file. Use this command syntax to clear these keys:
ssh-keygen -f ''<nowiki/>'/etc/ssh/ssh_known_hosts''' -R 'XXX.XXX.XXX.XXX'
ssh-keygen -f ''<nowiki/>'/etc/ssh/ssh_known_hosts''' -R 'node2.domain.com'
Now attempt to SSH into node2 again. You will get a nearly identical error, only this time the path to the known_hosts file will be in the root user's .ssh directory as so:
ssh-keygen -f ''<nowiki/>'/root/.ssh/ssh_known_hosts''' -R 'XXX.XXX.XXX.XXX'
ssh-keygen -f ''<nowiki/>'/root/.ssh/ssh_known_hosts''' -R 'node2.domain.com'
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|This second series of commands, pointing to /root/, are ''not replicated'' across the cluster members, and must be done on all members of the cluster manually.
|}
After executing these commands, when we join the node to the cluster, SSH will work correctly.

=== Join node to cluster. ===

# Power on server and configure LOM such as Dell DRAC to use the correct IP address.
# Edit /etc/hostname and /etc/hosts to confirm hostname is correctly matched to previous install's hostname.
# Reboot and verify hostname and IP are correct.
# If the previous machine had a Proxmox license, apply it now.
# Validate network connectivty on corosync network and both the Ceph frontend (consumption and management) and backend (replication) networks to all other nodes.
# Join the Proxmox cluster.
# Install Ceph.
# Add Ceph monitor and Ceph manager to this node.
# Migrate a test VM to the new node to confirm consumption.
# If there are any other maintenance tasks to complete (like swapping another node with the previous node's hardware) do NOT add OSDs back to node2 until ready.

Replacing A Proxmox Virtual Environment Server in a Ceph cluster

2024-12-24T19:43:45Z

Maeve:

== The Situation ==
Imagine a situation in which you have a cluster of Proxmox VE nodes as part of a hpyerconverged Ceph installation. You want to replace one of your nodes with a fully brand new installation of Proxmox VE. You don't intend to migrate its drives, you want to replace the node outright with new drives and a new installation of Proxmox VE. In order for this to work, there are a handful of things you need to consider to prevent 1. unnecessary Ceph rebalances 2. broken HA in Proxmox VE. The broken HA issue comes from the fact that Proxmox VE uses SSH internally to move data around and to run commands on remote nodes. These keys get messed up because Proxmox does not remove the keys when decommissioning a node.

== The Strategy ==
Migrate all virtual machines from the node. Ensure that you have made copies of any local data or backups that you want to keep. In addition, make sure to remove any scheduled replication jobs to the node to be removed.
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|Failure to remove replication jobs to a node before removing said node will result in the replication job becoming irremovable. Especially note that replication automatically switches direction if a replicated VM is migrated, so by migrating a replicated VM from a node to be deleted, replication jobs will be set up to that node automatically.
|}
Replacing a Proxmox Virtual Environment Server in hyperconverged Ceph configuration.

This guide assumes a four node cluster with hostnames node1-node4. We're assuming a replacement of node2, and for convenience, we're migrating to node1, but it could be to any other node in the cluster. In this guide, we are ''specifically'' replacing node2 with a new installation of PVE on an upgraded server that shares the same IP(s) and hostname, not migrating an existing installation to a new chassis.

=== Confirm cluster status (prerequisite, optional). ===
From node2's CLI, run ''pvecm nodes.''
node2# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node1
2 1 node2 (local)
3 1 node3
4 1 node4
Confirm that the node you intend to decommission is listed has the ''"(local)"'' identifier next to its name. This confirms we're on the right machine.

=== Migrate VM, CT, templates, storage off of node2. ===
Using HA, migrate running VM's from node2 to node1.

Manually migrate all remaining nonrunning VMs, CTs, and templates off of node2 to node1.

=== Decommission node from CEPH. ===

# Set OSDs on node2 to "out" and wait for rebalance. [[File:Ceph osd management.png]]
#* On Node2, go to Ceph -> OSD -> Node2 -> per each OSD listed, select the OSD and press "Out" in the top right of the control bar.
# Following rebalance, stop and destroy all OSDs on node2.
#* In the same screen of the PVE gui, select each out OSD and press stop, and once stopped, under the "more" submenu, select "destroy".
# Remove the Ceph servers (monitor, manager, and metadata) from node2.
#* In Ceph -> Monitor, select node2's monitor and press stop, and then once stopped, press destroy. [[File:Ceph mon.png]]
#* In Ceph -> Monitor, select node2's manager in the bottom panel, press stop, and once stopped, press destroy. [[File:Ceph manager.png]]
#* In Ceph -> CephFS - Metadata Servers, stop and destroy the Metadata Server for node2. [[File:Ceph mds.png]]
# On node2's CLI, clean up the Ceph CRUSH map and remove the host bucket using "ceph osd crush remove node2".
# From node1's CLI, run "pvecm delnode node2"

{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|''Important: Power off node2 before running pvecm delnode''. This is because the SSH keys and some internal references to the cluster may still exist on the node being removed, and it may attempt to perform sync operations within corosync and any distributed storage (i.e. Replicated ZFS pools).
|}
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|At this point, it is possible that you will receive an error message stating <code>Could not kill node (error = CS_ERR_NOT_EXIST)</code>. This does not signify an actual failure in the deletion of the node, but rather a failure in corosync trying to kill an offline node. Thus, it can be safely ignored.
|}

=== Confirm node deleted. ===
From node1's CLI, run ''pvecm nodes.''
node2# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node1 (local)
2 1 node3
3 1 node4

=== Clean up SSH Keys. ===
There is a '''major missing detail''' in the official documentation which is that if you intend to join a node into a cluster with the '''same hostname and IP''' the previous node had, everything will break if you don't take some prerequisite actions.

Power on the new node2 preconfigured with the same hostname and IP address.

From node1, SSH into the new node2. This will generate an error from the SSH client which will contain a line containing a command that will remove the key from the known hosts file. Use this command syntax to clear these keys:
ssh-keygen -f ''<nowiki/>'/etc/ssh/ssh_known_hosts''' -R 'XXX.XXX.XXX.XXX'
ssh-keygen -f ''<nowiki/>'/etc/ssh/ssh_known_hosts''' -R 'node2.domain.com'
Now attempt to SSH into node2 again. You will get a nearly identical error, only this time the path to the known_hosts file will be in the root user's .ssh directory as so:
ssh-keygen -f ''<nowiki/>'/root/.ssh/ssh_known_hosts''' -R 'XXX.XXX.XXX.XXX'
ssh-keygen -f ''<nowiki/>'/root/.ssh/ssh_known_hosts''' -R 'node2.domain.com'
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|This second series of commands, pointing to /root/, are ''not replicated'' across the cluster members, and must be done on all members of the cluster manually.
|}
After executing these commands, when we join the node to the cluster, SSH will work correctly.

=== Join node to cluster. ===

# Power on server and configure LOM such as Dell DRAC to use the correct IP address.
# Edit /etc/hostname and /etc/hosts to confirm hostname is correctly matched to previous install's hostname.
# Reboot and verify hostname and IP are correct.
# If the previous machine had a Proxmox license, apply it now.
# Validate network connectivty on corosync network and both the Ceph frontend (consumption and management) and backend (replication) networks to all other nodes.
# Join the Proxmox cluster.
# Install Ceph.
# Add Ceph monitor and Ceph manager to this node.
# Migrate a test VM to the new node to confirm consumption.
# If there are any other maintenance tasks to complete (like swapping another node with the previous node's hardware) do NOT add OSDs back to node2 until ready.

File:Ceph mds.png

2024-12-24T19:15:42Z

Maeve:

ceph_mds

File:Ceph manager.png

2024-12-24T19:13:47Z

Maeve:

ceph_manager

File:Ceph mon.png

2024-12-24T19:13:10Z

Maeve:

ceph_mon

File:Ceph osd management.png

2024-12-24T19:02:09Z

Maeve:

ceph_osd_management

Replacing A Proxmox Virtual Environment Server in a Ceph cluster

2024-12-24T16:39:30Z

Maeve:

Migrate all virtual machines from the node. Ensure that you have made copies of any local data or backups that you want to keep. In addition, make sure to remove any scheduled replication jobs to the node to be removed.
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|Failure to remove replication jobs to a node before removing said node will result in the replication job becoming irremovable. Especially note that replication automatically switches direction if a replicated VM is migrated, so by migrating a replicated VM from a node to be deleted, replication jobs will be set up to that node automatically.
|}
Replacing a Proxmox Virtual Environment Server in hyperconverged ceph configuration.

This guide assumes a four node cluster with hostnames node1-node4. We're assuming a replacement of node2, and for convenience, we're migrating to node1, but it could be to any other node in the cluster. In this guide, we are ''specifically'' replacing node2 with a new installation of PVE on an upgraded server that shares the same IP(s) and hostname, not migrating an existing installation to a new chassis.

=== Confirm cluster status (prerequisite, optional). ===
From node2's CLI, run ''pvecm nodes.''
node2# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node1
2 1 node2 (local)
3 1 node3
4 1 node4
Confirm that the node you intend to decommission is listed has the ''"(local)"'' identifier next to its name. This confirms we're on the right machine.

=== Migrate VM, CT, templates, storage off of node2. ===
Using HA, migrate running VM's from node2 to node1.

Manually migrate all remaining nonrunning VMs, CTs, and templates off of node2 to node1.

=== Decommission node from CEPH. ===

# Set OSDs on node2 to "out" and wait for rebalance.
#* On Node2, go to Ceph -> OSD -> Node2 -> per each OSD listed, select the OSD and press "Out" in the top right of the control bar.
# Following rebalance, stop and destroy all OSDs on node2.
#* In the same screen of the PVE gui, select each out OSD and press stop, and once stopped, under the "more" submenu, select "destroy".
# Remove the Ceph servers (monitor, manager, and metadata) from node2.
#* In Ceph -> Monitor, select node2's monitor and press stop, and then once stopped, press destroy.
#* In Ceph -> Monitor, select node2's manager in the bottom panel, press stop, and once stopped, press destroy.
#* In Ceph -> CephFS - Metadata Servers, stop and destroy the Metadata Server for node2.
# On node2's CLI, clean up the Ceph CRUSH map and remove the host bucket using "ceph osd crush remove node2".
# From node1's CLI, run "pvecm delnode node2"

{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|''Important: Power off node2 before running pvecm delnode''. This is because the SSH keys and some internal references to the cluster may still exist on the node being removed, and it may attempt to perform sync operations within corosync and any distributed storage (i.e. Replicated ZFS pools).
|}
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|At this point, it is possible that you will receive an error message stating <code>Could not kill node (error = CS_ERR_NOT_EXIST)</code>. This does not signify an actual failure in the deletion of the node, but rather a failure in corosync trying to kill an offline node. Thus, it can be safely ignored.
|}

=== Confirm node deleted. ===
From node1's CLI, run ''pvecm nodes.''
node2# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node1 (local)
2 1 node3
3 1 node4

=== Clean up SSH Keys. ===
There is a major missing detail in the official documentation which is that if you intend to join a node into a cluster with the same hostname and IP the previous node had, everything will break if you don't take some prerequisite actions.

Power on the new node2 preconfigured with the same hostname and IP address.

From node1, SSH into the new node2. This will generate an error from the SSH client which will contain a line containing a command that will remove the key from the known hosts file. Use this command syntax to clear these keys:
ssh-keygen -f ''<nowiki/>'/etc/ssh/ssh_known_hosts''' -R 'XXX.XXX.XXX.XXX'
ssh-keygen -f ''<nowiki/>'/etc/ssh/ssh_known_hosts''' -R 'node2.domain.com'
Now attempt to SSH into node2 again. You will get a nearly identical error, only this time the path to the known_hosts file will be in the root user's .ssh directory as so:
ssh-keygen -f ''<nowiki/>'/root/.ssh/ssh_known_hosts''' -R 'XXX.XXX.XXX.XXX'
ssh-keygen -f ''<nowiki/>'/root/.ssh/ssh_known_hosts''' -R 'node2.domain.com'
{|
|[[File:Warning-sign-icon-transparent-background-free-png.webp|60x60px]]
|This second series of commands, pointing to /root/, are ''not replicated'' across the cluster members, and must be done on all members of the cluster manually.
|}
After executing these commands, when we join the node to the cluster, SSH will work correctly.

=== Join node to cluster. ===

# Power on server and configure LOM such as Dell DRAC to use the correct IP address.
# Edit /etc/hostname and /etc/hosts to confirm hostname is correctly matched to previous install's hostname.
# Reboot and verify hostname and IP are correct.
# If the previous machine had a Proxmox license, apply it now.
# Validate network connectivty on corosync network and both the Ceph frontend (consumption and management) and backend (replication) networks to all other nodes.
# Join the Proxmox cluster.
# Install Ceph.
# Add Ceph monitor and Ceph manager to this node.
# Migrate a test VM to the new node to confirm consumption.
# If there are any other maintenance tasks to complete (like swapping another node with the previous node's hardware) do NOT add OSDs back to node2 until ready.

File:Warning-sign-icon-transparent-background-free-png.webp

2024-12-24T15:34:44Z

Maeve:

Replacing A Proxmox Virtual Environment Server in a Ceph cluster

2024-12-24T02:09:11Z

Maeve: Created page with "Move all virtual machines from the node. Ensure that you have made copies of any local data or backups that you want to keep. In addition, make sure to remove any scheduled replication jobs to the node to be removed. {| class="wikitable" | |Failure to remove replication jobs to a node before removing said node will result in the replication job becoming irremovable. Especially note that replication automatically switches direction if a replicated VM is migrated, so by mi..."

Move all virtual machines from the node. Ensure that you have made copies of any local data or backups that you want to keep. In addition, make sure to remove any scheduled replication jobs to the node to be removed.
{| class="wikitable"
|
|Failure to remove replication jobs to a node before removing said node will result in the replication job becoming irremovable. Especially note that replication automatically switches direction if a replicated VM is migrated, so by migrating a replicated VM from a node to be deleted, replication jobs will be set up to that node automatically.
|}
Replacing a Proxmox Virtual Environment Server in hyperconverged ceph configuration.

This guide assumes a four node cluster with hostnames node1-node4. We're assuming a replacement of node2, and for convenience, we're migrating to node1, but it could be to any other node in the cluster. In this guide, we are ''specifically'' replacing node2 with a new installation of PVE on an upgraded server that shares the same IP(s) and hostname, not migrating an existing installation to a new chassis.

=== Migrate VM, CT, templates, storage off of node2. ===
Using HA, migrate running VM's from node2 to node1.

Manually migrate all remaining nonrunning VMs, CTs, and templates off of node2 to node1.

=== Decommission node from CEPH ===
Set OSDs on node2 to "out" and wait for rebalance.

On Node2, go to Ceph -> OSD -> Node2 -> per each OSD listed, select the OSD and press "Out" in the top right of the control bar.

Following rebalance, stop and destroy all OSDs on node2.

In the same screen of the PVE gui, select each out OSD and press stop, and once stopped, under the "more" submenu, select "destroy".

Remove the Ceph servers (monitor, manager, and metadata) from node2.

In Ceph -> Monitor, select node2's monitor and press stop, and then once stopped, press destroy.

In Ceph -> Monitor, select node2's manager in the bottom panel, press stop, and once stopped, press destroy.

In Ceph -> CephFS - Metadata Servers, stop and destroy the Metadata Server for node2.

On node2's CLI, clean up the Ceph CRUSH map and remove the host bucket using "ceph osd crush remove node2".

''!! Important: Power off node2 before running pvecm delnode''. This is because the SSH keys and some internal references to the cluster may still exist on the node being removed, and it may attempt to perform sync operations within corosync and any distributed storage (i.e. Replicated ZFS pools).

From node1's CLI, run "pvecm delnode node2".

=== Clean up SSH Keys ===
Power on the new node2 preconfigured with the same hostname and IP address.

From node1, SSH into the new node2. This will generate an error from the SSH client which will contain a line containing a command that will remove the key from the known hosts file. Use this command syntax to clear these keys:

'' ssh-keygen -f '/etc/ssh/ssh_known_hosts' -R 'XXX.XXX.XXX.XXX'''

'' ssh-keygen -f '/etc/ssh/ssh_known_hosts' -R 'node2.domain.com'''

Now attempt to SSH into node2 again. You will get a nearly identical error, only this time the path to the known_hosts file will be in the root user's .ssh directory as so:

'' ssh-keygen -f '/root/.ssh/ssh_known_hosts' -R 'XXX.XXX.XXX.XXX'''

'' ssh-keygen -f '/root/.ssh/ssh_known_hosts' -R 'node2.domain.com'''

!! This second series of commands, pointing to root, are ''not replicated'' across the cluster members, and must be done on all members of the cluster manually.

In the following example, we will remove the node hp4 from the cluster.

Log in to a '''different''' cluster node (not hp4), and issue a <code>pvecm nodes</code> command to identify the node ID to remove:
<code>hp1# pvecm nodes

Membership information
<nowiki>~~~~~~~~~~~~~~~~~~~~~~</nowiki>
Nodeid Votes Name
1 1 hp1 (local)
2 1 hp2
3 1 hp3
4 1 hp4</code>
At this point, you must power off hp4 and ensure that it will not power on again (in the network) with its current configuration.
{| class="wikitable"
|
|As mentioned above, it is critical to power off the node '''before''' removal, and make sure that it will '''not''' power on again (in the existing cluster network) with its current configuration. If you power on the node as it is, the cluster could end up broken, and it could be difficult to restore it to a functioning state.
|}
After powering off the node hp4, we can safely remove it from the cluster.
<code>hp1# pvecm delnode hp4
Killing node 4</code>
{| class="wikitable"
|
|At this point, it is possible that you will receive an error message stating <code>Could not kill node (error = CS_ERR_NOT_EXIST)</code>. This does not signify an actual failure in the deletion of the node, but rather a failure in corosync trying to kill an offline node. Thus, it can be safely ignored.
|}
Use <code>pvecm nodes</code> or <code>pvecm status</code> to check the node list again. It should look something like:
<code>hp1# pvecm status

...

Votequorum information
<nowiki>~~~~~~~~~~~~~~~~~~~~~~</nowiki>
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
<nowiki>~~~~~~~~~~~~~~~~~~~~~~</nowiki>
Nodeid Votes Name
0x00000001 1 192.168.15.90 (local)
0x00000002 1 192.168.15.91
0x00000003 1 192.168.15.92</code>
If, for whatever reason, you want this server to join the same cluster again, you have to:

* do a fresh install of Proxmox VE on it,
* then join it, as explained in the previous section.

The configuration files for the removed node will still reside in ''/etc/pve/nodes/hp4''. Recover any configuration you still need and remove the directory afterwards.

Proxmox Host SSH keys

2024-12-13T18:46:12Z

Maeve:

== Intended method: ==
Delete old ssh host keys:
rm /etc/ssh/ssh_host_*
Reconfigure OpenSSH Server:
dpkg-reconfigure openssh-server
Update all ssh client(s) at ~/.ssh/known_hosts

Then update certs and keys ''on each machine'':
pvecm updatecerts -f

== Manual method<ref>https://forum.proxmox.com/threads/pvecm-updatecert-f-not-working.135812/page-3#post-660500</ref>: ==
If this fails (which it might), log into each troublesome node through SSHd and copy the public key from
/etc/ssh/ssh_host_rsa_key.pub.
Copy this to
/etc/pve/nodes/<node>/ssh_known_hosts
and prepend it with that machine's hostname. Assuming a hostname of pve1, this line should appear as
pve1 ssh-rsa <key>
Then restart the SSH daemon:
systemctl restart sshd

Category:Linux Tutorials

2024-12-11T00:17:08Z

Maeve: /* Linux Tutorials (Especially helpful for Proxmox) */

Here are some of our Linux tutorials.

== Linux Tutorials (Especially helpful for Proxmox) ==

* [[Offline Uncorrectable Sectors]]
* [[ZFS Failed Disk Replacement]]
* [[Calculating SSD Wearout]]
* [[Proxmox Backup Server Replication]]
* [[Proxmox Host SSH keys]]
* [[Replacing Proxmox Virtual Environment Server in a Ceph cluster]]

Replacing Proxmox Virtual Environment Server in a Ceph cluster

2024-12-11T00:17:00Z

Maeve: Created page with "Replacing a Proxmox Virtual Environment Server in hyperconverged ceph configuration. This guide assumes a four node cluster with hostnames node1-node4. We're assuming a replacement of node2. # Using HA, migrate running VM's from node2 to node1, or any other location that has ample resources and is currently a member of CEPH. # Set OSDs on node2 to "out" and wait for rebalance. # Following rebalance, stop and destroy all OSDs on node2. # Remove Ceph mon and manager fro..."

Replacing a Proxmox Virtual Environment Server in hyperconverged ceph configuration.

This guide assumes a four node cluster with hostnames node1-node4. We're assuming a replacement of node2.

# Using HA, migrate running VM's from node2 to node1, or any other location that has ample resources and is currently a member of CEPH.
# Set OSDs on node2 to "out" and wait for rebalance.
# Following rebalance, stop and destroy all OSDs on node2.
# Remove Ceph mon and manager from node2.
# Clean up the Ceph CRUSH map and remove the host bucket using "ceph osd crush remove node2".
# From a node that's still participating in Ceph, run "pvecm delnode node2".

The node is now decommissioned and no longer participating in Ceph. It can be removed. Let's install the replacement.

# Physically remove old server, install new server. Cable and power server.
# Configure LOM such as Dell DRAC to use the correct IP address.
# Set new node2's IP management IP address to the IP of the previous machine. Validate connectivity.
# Edit /etc/hostname and /etc/hosts to confirm hostname is correctly matched to previous install's hostname.
# Reboot and verify hostname and IP are correct.
# If the previous machine had a Proxmox license, apply it now.
# Validate network connectivty on corosync network and both the Ceph frontend (consumption and management) and backend (replication) networks to all other nodes.
# Join the Proxmox cluster.
# Install Ceph.
# Add Ceph Mon and Ceph Manager to this node.
# Migrate a test VM to the new node to confirm consumption.
# If there are any other maintenance tasks to complete (like swapping another node with the previous node's hardware) do NOT add OSDs back to node2 until ready.

A similar series of steps can be taken if existing drives are being moved to a new server intallation, maintaining OS and OSDs, as opposed to new drives. We'll assume a replacement of node 1 with the previous node2's. hardware.

# Using HA, migrate running VM's from node1 to node2, or any other location that has ample resources and is currently a member of CEPH.
# Unlike before, set the noout flag - the OSDs aren't actually going anywhere, so we do not want a rebalancing.
# Shutdown node1.
# Physially move boot and data drives from node1 to the donor that was previously node2.
# Un-rack now driveless node1 and replace with now populated donor node2. Cable.
# Power on and configure LOM such as Dell DRAC to use the correct IP address.
# Validate system boot.
# Validate network connectivty on corosync network and both the Ceph frontend (consumption and management) and backend (replication) networks to all other nodes.
# Verify that OSDs are online and that all PGs report synced.
# Disable noout flag.
# We can now safely add OSDs back to the new node2 and allow it to rebalance. This could take a large amount of time - up to several days, depending on quantity of storage.

ZFS Remote Sync

2024-11-25T20:51:35Z

Maeve: Created page with "It is possible to '''sync one zpool's contents to another''' via ''zfs send'' and ''zfs recv.'' First, create a snapshot of the data you want to send. To copy an entire zpool, create a snapshot with the following command: zpool snapshot -r [zpool_to_snapshot]@[snapshot_name] You can also snapshot a specific subvol. zpool snapshot [zpool_to_snapshot/with_sub_vol]@[snapshot_name] We'll be sending the data from this snapshot over to another host using zfs send and zfs..."

It is possible to '''sync one zpool's contents to another''' via ''zfs send'' and ''zfs recv.''

First, create a snapshot of the data you want to send. To copy an entire zpool, create a snapshot with the following command:
zpool snapshot -r [zpool_to_snapshot]@[snapshot_name]
You can also snapshot a specific subvol.
zpool snapshot [zpool_to_snapshot/with_sub_vol]@[snapshot_name]
We'll be sending the data from this snapshot over to another host using zfs send and zfs recv. We'll want to run this command in a screen session in case our terminal disconnects. We pipe the output of zfs send into an ssh connection running zfs recv.

We run zfs send with -R to send all content under the specified snapshot. We run zfs recv with the flags -F (expand and replace) -d (discard target name and replace with source) -u (do not mount on destination). The pool must exist on the destination. Essentially, we're not cloning the ZFS configuration, we're funneling information on a higher level between them, in the same sense that one might SCP or rsync data. We recommend using zpool set autoexpand=on [pool_name]
zfs send -R [zpool_to_clone]@[snapshot_name] | ssh [destination_IP] zfs recv -Fdu [destination_pool]
This will take quite some time. It may take multiple days, depending on the size of your dataset.

Category:Linux Tutorials

2024-10-31T18:34:19Z

Maeve: /* Linux Tutorials (Especially helpful for Proxmox) */

Packetfence

2024-10-28T11:42:20Z

Maeve:

'''Packetfence''' is an open source '''Network Access Control''' solution developed by Inverse Inc. It provides a RADIUS server that pools together authentication sources from Active Directory, the Google suite, HTPasswd, another RADIUS server, generic LDAP, etc. It also has several supported methods, through its RADIUS server, of allowing authentication to client devices, including dot1x.

== Installing Packetfence ==
A Debian-based installation ISO for Packetfence is [https://us-ord-1.linodeobjects.com/packetfence-iso/v14.0.0/PacketFence-ISO-v14.0.0.iso available for download here.] The stable release is recommended. This release comes with all of its main services pre-installed and given sane defaults. A minimum requirement of 200GB of storage and 16GB of ram. It is recommended to use multiple network interfaces, one for management and several for other purposes, but for documentation's sake we'll start with one network interface.

# Set the hostname.
# Specify the domain name if not autopopulated through DHCP.
# Set the root password.
# After setting the root password, the installer will take some time. It may appear stuck at some points, but it just takes a while. Up to an hour. '''Be patient.'''

=== UEFI fix ===
When you install Packetfence on Proxmox using the standard recommended configuration settings (Q35 machine type, OVMF bios) will result in some issues after the machine reboots. You'll be prompted with a yellow and black UEFI shell, requiring you to run ''fs0:\efi\debian\grubx64.efi'' every time the machine boots, which is obviously not going to work if the host loses power.

To fix this, on the first boot of this machine<ref>[https://forum.proxmox.com/threads/how-to-boot-grubx64-efi-after-import-from-hyper-v.55429/#post-255178 Paraphrased from this Proxmox Forum post] </ref>:

# Press ESC immediately to get into the BIOS.
# Go to 'Boot Maintenance Manager'
# Go to 'Boot Option'
# Go to 'Add Boot Option'
# Press enter on the 'PciRoot' volume
# Select EFI and press enter
# Select the debian folder.
# Select grubx64.efi and press enter.
# Enter "Boot into Packetfence" as the description.
# Press F10 to save again.
# Go to 'Commit Changes and Exit'
# Select 'Change Boot Order'
# Press enter to get the listing of boot devices.
# Go to 'Boot into Packetfence'
# Press + until this boot option is on top of the list and press enter
# Press F10 to save (just to be sure)
# Select 'Commit Changes and Exit'

== First time configuration ==
Packetfence most likely received a DHCP lease from your DHCP server. You can either check your DHCP server or run ip addr to determine the default IP address. Assuming a DHCP lease of ''192.168.1.128,'' access the management interface from <nowiki>https://192.168.1.128:1443/</nowiki>. The first prompt will be a listing of network interfaces. Click the interface and it will take you to a submenu where you can specify the network settings for management. This will take effect immediately so you will need to navigate to the new address. The first few prompts are extremely straight forward, generating the admin user account and password, and selecting the management interface. Set your domain and hostname if you wish to change them from what was set during Debian installation. Also specify a timezone and log recipients.

Under the log recipients section, we recommend using the advanced settings context in the top right corner of the settings card so you can specify your SMTP server host settings.

You can skip the Fingerbank setting if you don't intend to use it.

Save the default passwords generated for the internal services somewhere safe as they can neither be retrieved again or altered. Reboot the machine after the configuration is done, for good measure. Not every source gets properly restarted after the wizard concludes.

== Authentication and authorization flow preamble ==
This is where things get murky with the official documentation lacking clarity and specificity.

The flow of behavior is as follows:

# Device connects through Packetfence client device like a Cisco switch - this uses EAP between the port on the device and the NIC of the client machine.
# The switch then has a configuration that tells it to communicate to the Packetfence server using RADIUS, providing Packetfence with the credentials the user gave over EAP.
# Packetfence acts a grand arbiter between all of your unique authentication sources and configurations (which is what we will configure below) and determines what to tell the switch to do with that user's port. This response is over RADIUS.
# Once the user has authenticated, the port they're connected to is assigned a VLAN, which we will assume you have a properly configured DHCP server waiting on. Once Packetfence has taken care of the user's authentication and set the VLAN, the rest of your networking infrastructure is unchanged to accommodate the user.

The first thing we need to define is the connection point between your Packetfence server and your Active Directory Domain Controller.

== Active Directory Domain Connection ==
Go to Configuration -> Policies and Access Control -> Active Directory Domains.

Press "New Domain".

* Identifier is the name used throughout the rest of the Packetfence gui referencing this domain configuration,
* Workgroup can be set to the name of the domain.
* Set DNS name to the FQDN of your active directory domain.
* Sticky DC can be left as *
* Active Directory FQDN is the FQDN of the domain controller you're going to make queries against.
* Active Directory IP and DNS Server(s) can both be set to the IP of your domain controller.
* OU should be set to Computers or potentially something more specific. This is the OU that Packetfence will create its machine account under in AD.
* The last two fields are Domain Administrator Username and Password. We recommend having a dedicated admin account for this, just to separate concerns.
** Note that these credentials are only used when generating the machine account and storing its hashed password, and will not store these credentials permanently.
* Set Allow on Registration to true.

After saving, return to the previous menu. You should see a green light under "domain joined" and a 200 HTTP status code in the top right corner.

Packetfence can now communicate to Active Directory. However at this stage, it's not been configured to actually ''use'' this communication. Let's add an Active Directory Authentication Source.

== Active Directory Authentication Source ==
From the same configuration tab, let's go to Authentication Sources.

Press "New Internal Source" and select Active Directory from the drop down.

* Give it a name, preferably that includes the domain itself, as well as a description.
* Host should be set to the domain controller you specified in the previous step.
* Base DN should be set to a string of the format "dc=domain,dc=local,dc=example,dc=com" - take the FQDN of your domain and take each element and prefix it with dc=. Use commas between them.
* Bind DN is going to be set with a similar string detailing which user account Packetfence binds with. If you created your Packetfence admin in Active Directory under the Users OU, this will be something of the form "cn=packetfence,cn=users,dc=domain,dc=local,dc=example,dc=com".
* Provide the password below.
* Set "associated realms" to ''default'' and ''null''.
* Create an '''Authentication Rule:'''
** Name: catchall
** Description: catchall
** Matches: any
** Conditions: none
** Actions:
*** Role: default
*** Access Duration: 12 hours
* Save the source.

Now, when we configure our RADIUS client device, such as an AP or a switch, credentials will be acceptable from Active Directory, and authenticated users will be given 12 hours of access on a default VLAN.

== Configuring the switch (Catalyst 2960) ==
The rest of the packetfence installation guides assumes a Catalyst 2960. Most modern IOS devices use the same syntax so we'll use their template.
dot1x system-auth-control
aaa new-model
aaa group server radius packetfence
server PF_MANAGEMENT_IP auth-port 1812 acct-port 1813
aaa authentication login default local
aaa authentication dot1x default group packetfence
aaa authorization network default group packetfence
radius-server host PF_MANAGEMENT_IP auth-port 1812 acct-port 1813 timeout 2 key useStrongerSecret
radius-server vsa send authentication
snmp-server community public RO
snmp-server community private RW
Then on a port we wish to use dot1x on:
authentication host-mode single-host
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer restart 10800
authentication timer reauthenticate 10800
mab
no snmp trap link-status
dot1x pae authenticator
dot1x timeout quiet-period 2
dot1x timeout tx-period 3

== Adding switch to Packetfence ==
We then navigate to Configuration -> Policies and Access Control -> Network Devices -> Switches.

Press "New Switch" and select default when prompted.

Enter the management IP of the switch and select "production".

Under Type, select Cisco Catalyst 2960.

In the Radius tab, enter the secret specified in the steps above.

Make sure VLAN by Role ID is enabled and that "default" has been set to a VLAN on your network that you wish to grant access to.

== Connection profile ==
Lastly, let's go to Configuration -> Policies and Access Control -> Connection Profiles.

Click on "New Connection Profile".

* Profile name: 8021x.
* Profile description: 8021x wired connections
* Enable the profile.
* Automatically register devices: checked
* FIlters:
** Match any
** Connection Type: Ethernet EAP
* Add the ADDC source we created earlier

You can now test your workstation.

Packetfence

2024-10-24T23:30:19Z

Maeve: Wrote all the introduction and clarification on booting issue

Category:Software Tutorials

2024-10-17T12:45:24Z

Maeve: Added packetfence link

Here are some of our tutorials and notes on software, both selfhosted FOSS and 3rd party products.

== Software Tutorials ==

* [[Firebox Content Inspection|Firebox Content Inspection (HTTPS Content Inspection)]]
* [[Nextcloud]]
* [[Packetfence]]
* [[Proxy Server]] (high level)
** [[Reverse Proxy]] (use-case specific)
* [[Zabbix]]

Category:Networking Tutorials

2024-10-14T16:40:45Z

Maeve: /* Switch configuration[1] [2] */

Here are some of our networking related guides.

== Web Hosting ==

=== Proxying requests ===

* [[Proxy Server]]
* [[Reverse Proxy]]

=== Switch configuration<ref>[[:Category:Catalyst Tutorials|Catalyst Tutorials]]</ref> <ref>[[:Category:Nexus Tutorials|Nexus Templates]]</ref> ===

* [[Cisco Catalyst Template]]
* [[Cisco Catalyst SNMP Tutorial]]
* [[Cisco Nexus Template]]

Category:Catalyst Tutorials

2024-10-14T16:39:55Z

Maeve: /* General templates */

Below is a collection of related configuration snippets for Catalyst switches that we've collected for certain purposes over the years. We've generalized and anonymized them to be readily available as templates for your needs.

== General templates ==

* [[Cisco Catalyst Template]]

== Specific features ==

* [[Cisco Catalyst SNMP Tutorial|Cisco Catalyst SNMP]]

Cisco Catalyst SNMP Tutorial

2024-10-14T16:23:41Z

Maeve:

'''SNMP''' is a protocol built into most network devices (or available through installable packages on Linux, BSD) that enables authenticated remote monitoring of a device without CLI access.

== Creating SNMP Views ==
The first thing we're going to do is create an SNMP View. This is essentially an access level indicator that can be supplied as an argument when creating a group. This allows us to specify which items in SNMP's MIB Database we want our users to access. It also allows us to specify read/write permissions.
snmp-server view Zabbix iso included
Here, we enter the snmp-server command context, then we define a view named Zabbix. Then, after the name, we set the MIB or OID name that we want to target - iso is the global SNMP namespace, it includes everything SNMP itself records - and then we say that this view is inclusive, meaning anything under iso is to be included in the view.

Now, let's create another one, but this time, it'll prevent access instead.
snmp-server view Zabbix_ReadOnly iso excluded
Now, we have a second view called Zabbix_ReadOnly that can't access ''anything''. This will be our write permissions for Zabbix.

== Creating SNMP Groups ==
Let's create a group for our SNMPv3 user to be a part of.
snmp-server group zbx v3 priv read Zabbix write Zabbix_ReadOnly
Here we specify that for proper communication with SNMP, we need to use the authPriv user level which requires two different passwords, one for user auth and one for encryption.

Groups can have three different views specified.

* Read view defines permissions for standard read operations
* Write views define permissions for management - some settings can be controlled through SNMP, but Zabbix does not require write privileges.
* Notify views define access for SNMP users when sending traps and informs. We're not specifically setting traps up in this guide so our notify view isn't set

== Creating an SNMP User ==
Now we can create an SNMPv3 user.
snmp-server user zabbix zbx v3 auth sha '''strongerAuthPassword''' priv aes 256 '''strongerPrivPassword'''
Now, if we run ''show snmp user:''
User name: zabbix
Engine ID: 800000090300848A8DEC9A00
storage-type: nonvolatile active
Authentication Protocol: SHA
Privacy Protocol: AES256
Group-name: zbx
We can see that the user has been added.
[[Category:SNMP]]
[[Category:Networking Tutorials]]
[[Category:Catalyst Tutorials]]

Cisco Catalyst SNMP Tutorial

2024-10-14T16:20:26Z

Maeve: Created page with "'''SNMP''' is a protocol built into most network devices (or available through installable packages on Linux, BSD) that enables authenticated remote monitoring of a device without CLI access. == Creating SNMP Views == The first thing we're going to do is create an SNMP View. This is essentially an access level indicator that can be supplied as an argument when creating a group. This allows us to specify which items in SNMP's MIB Database we want our users to access. It..."

Calculating SSD Wearout

2024-10-11T13:25:21Z

Maeve: Created page with "'''SSDs have a predetermined wearout''' point at which they enter a failed state. This is because SSDs use nand flash which eventually loses its ability to accurate respond to IO requests. Determining the status of your SSD is vital, as knowing the state of its wearout can help you schedule much needed replacements. First, you need to know the drive's ''rated'' '''TBW''' - '''terabytes or total bytes written -''' as this is what we'll be calculating against. This repre..."

'''SSDs have a predetermined wearout''' point at which they enter a failed state. This is because SSDs use nand flash which eventually loses its ability to accurate respond to IO requests. Determining the status of your SSD is vital, as knowing the state of its wearout can help you schedule much needed replacements.

First, you need to know the drive's ''rated'' '''TBW''' - '''terabytes or total bytes written -''' as this is what we'll be calculating against. This represents what the drive is rated to be able to handle. This can be determined usually through spec sheets or by using [https://www.techpowerup.com/ssd-specs/ this useful database.] It's not exhaustive, but it's usually good enough. For this example, our drive is rated for 400TB TBW.

Next, you need to know your drive's '''Total_LBAs_Written -''' a measurement of how many sectors have been written or modified on the drive. This can be determined by using the following:
root@example-pve:~# smartctl --all /dev/sdX
--- snippet ---
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 1
'''246 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 384578442888'''
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 12018275091
248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 84402429335
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 9380
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 4
The exact values available on your device differ from model to model. Some even have an explicit field for expected lifetime remaining, but not all do, so this method may be the best option.

Then, we multiply this value (on the right hand side, the raw bytes value) by the Sector Size of the drive. This can also be gotten from this command:
Sector Size: 512 bytes logical/physical
Now we can get the actual terabyte amount using this formula:
(total_lbas_written * sector_size) / (1024^4)
(384578442888 * 512) / (1024^4)
'''≈ 179TB'''
Now, we know that on this drive, '''179TB''' has been written. Remember how our drive is rated for about 400TB? That means that our drive currently has '''(179/400) ≈ .45 ≈ 45% wearout.'''

Proxmox uses a slightly more sophisticated internal algorithm, and may use a self-discovered max TBW rating indicator, which could explain discrepancies within a few percent. Case in point, Proxmox tells us that this drive actually has 49% wearout. But hey, that's really close! Close enough to schedule on.

Category:Linux Tutorials

2024-10-11T12:51:21Z

Maeve:

Here are some of our Linux tutorials.

== Linux Tutorials (Especially helpful for Proxmox) ==

* [[Offline Uncorrectable Sectors]]
* [[ZFS Failed Disk Replacement]]
* [[Calculating SSD Wearout]]

Category:Networking Tutorials

2024-10-10T23:11:38Z

Maeve:

Category:Nexus Tutorials

2024-10-10T23:10:12Z

Maeve: Created page with "Here are some of our tutorials and templates for the Nexus series of Layer 3 switches by Cisco. == Nexus Tutorials == * Cisco Nexus Template"

Here are some of our tutorials and templates for the Nexus series of Layer 3 switches by Cisco.

== Nexus Tutorials ==

* [[Cisco Nexus Template]]

Cisco Nexus Template

2024-10-10T23:08:33Z

Maeve: Created page with " ! Set up hostname, management hostname examplehostname ! Some optional features we assume you'll want no feature telnet feature scp-server feature interface-vlan feature dhcp feature lldp ! Management interface (Nexuses have dedicated gigabit ports for management interface mgmt0 vrf member management ip address 192.168.10.10/24 ! Here we define the router we want our management VRF to use as its gateway. vrf context management ip route 0.0..."

! Set up hostname, management
hostname examplehostname

! Some optional features we assume you'll want
no feature telnet
feature scp-server
feature interface-vlan
feature dhcp
feature lldp

! Management interface (Nexuses have dedicated gigabit ports for management
interface mgmt0
vrf member management
ip address 192.168.10.10/24

! Here we define the router we want our management VRF to use as its gateway.
vrf context management
ip route 0.0.0.0/0 192.168.10.1

! Set domain information
ip domain-lookup
ip domain-name your-local-domain-here
ip name-server 192.168.10.1 use-vrf management

! Setup Network Time Protocol
ntp server 0.north-america.pool.ntp.org
ntp server 1.north-america.pool.ntp.org
ntp server 2.north-america.pool.ntp.org
ntp server 3.north-america.pool.ntp.org

! This is optional, but if you're used to IOS' wr to save configs, this lets you use that same phrase.
cli alias name wr copy run start

! Set up users
username AdminUser password 5 supersecretadminpassword role network-admin

! Example vlan. Note that we're just setting a description.
vlan 10
name LAN_Management_192.168.10.0/24

! Example vlan interface
interface vlan 10
   description Management_Address
   ip address 192.168.10.2/24
   vrf member management
   no shutdown
   no ip redirects

! Enables management
line console
line vty
copy running-config startup-config

Category:Networking Tutorials

2024-10-10T17:36:57Z

Maeve:

Cisco Catalyst Template

2024-10-10T17:35:11Z

Maeve: Created entire page

! '''Enter configuration mode'''
en
conf t

! '''Turn on rapid spanning tree'''
spanning-tree mode rapid-pvst

! '''Set the hostname'''
hostname '''examplename'''

! '''Enables password encryption'''
service password-encryption

! '''Optional. Enable secret followed by a password will require console users to provide a password before they can "enable" the switch, allowing them to edit conf mode, run show run, etc'''
enable secret '''supersecretpassword'''

! '''Create a superuser / admin. The name can be anything'''.
username AdminUser priv 15 secret '''incrediblysecurepassword'''

! '''Set Timezone'''
clock timezone UTC 0 0

! '''Set NTP server. If DNS is functional, use one of these'''.
ntp server 0.north-america.pool.ntp.org
ntp server 1.north-america.pool.ntp.org
ntp server 2.north-america.pool.ntp.org
ntp server 3.north-america.pool.ntp.org
! '''Alternatively, if you want this device to instead pull NTP from another device in your network, supply an IP'''.
! ntp server '''10.0.0.1'''

! '''aaa new-model enables a suite of features that are now standard across all other Cisco devices.'''
aaa new-model
aaa authentication login default local
! '''Console connection'''
aaa authorization console
! '''SSH connection'''
aaa authorization exec default local

! '''Disables requirement for password on the console. If enabled, this would be one step behind enable secret.'''
line con 0
no password
exit

! '''This configures SSH access in the same way that we configured the above console connection. a vty is a remote connection. IOS supports 16 concurrently'''.
line vty 0 15
no password
transport input ssh
exit

! '''Disables default interface for VLAN 1. VLAN 1 should be avoided when possible as this is the default VLAN that ports will take when reset.'''
int vlan 1
no ip address
shutdown

! '''Define a default domain name.'''
!no ip domain-lookup
ip domain-name '''your-internal-domain-name'''
ip name-server '''your-local-dns-server'''
! Generates a cryptokey to enable SSH
crypto key generate rsa modulus 4096
ip ssh version 2
! ip ssh {timeout seconds | authentication-retries number}

! '''Here we add a port to VLAN 10. This assumes VLAN 10 is the VLAN you're going to use for your primary management. Adjust as appropriate.'''
int te 1/0/48
switchport access VLAN 10
int vlan 18
ip address 192.168.10.10 255.255.255.0
ip default-gateway 192.168.10.1
exit

! '''Ping the device from itself to ensure the interface has come alive.'''
ping 192.168.10.10

! '''Write to memory.'''
wr

Category:Catalyst Tutorials

2024-10-10T16:25:55Z

Maeve: Created page with "Below is a collection of related configuration snippets for Catalyst switches that we've collected for certain purposes over the years. We've generalized and anonymized them to be readily available as templates for your needs. == General templates == * Cisco Catalyst Template"

Main Page

2024-10-10T16:17:45Z

Maeve: /* Categories */

Welcome to the Rosemark Networks Consultants Wiki.

== Categories ==

* [[:Category:Windows Tutorials|Windows Tutorials]]
* [[:Category:Linux Tutorials|Linux Tutorials]]
* [[:Category:Software Tutorials|Software Tutorials]]
* [[:Category:Networking Tutorials|Networking Tutorials]]