Adding a Second PVE Node While Keeping Existing Guests Running
Adding a Second PVE Node While Keeping Existing Guests Running
The first PVE host already ran domain controllers, ASR, and monitoring. The second was a fresh empty installation. On September 19, 2026, they became a cluster for unified management and manual migration. No HA resources were configured, and QDevice was still a plan.
The important decisions were which host created the cluster and how to demonstrate that the existing guests remained running throughout.
Four separate responsibilities
| Component | Responsibility |
|---|---|
| PVE cluster / pmxcfs | Configuration distribution and management |
| Corosync / Knet | Membership and cluster communication |
| QDevice | External voting assistance |
| PVE HA | Failure recovery for explicitly managed resources |
Without configured HA resources, creating this cluster did not automatically relocate VMs. Identical local-lvm names also did not make the disks shared.
Create on the populated host; join the empty host
Joining replaces the joining node’s /etc/pve configuration:
pve (existing VM/LXC workloads) creates → pve2 (empty) joinsReversing this direction puts existing definitions at risk. Inspect qm list, pct list, virtual disks, old Corosync configuration, and active tasks. Do not override existing guests or cluster state with --force.
The installed version’s internal assert_joinable() check returned JOINABLE_OK. This supplemented the supported procedure; an internal Perl interface is not a stable public API.
Preflight the actual environment
The recorded PVE Manager versions were 9.2.2 and 9.2.20. Their Join API checks passed. This establishes the tested combination, not arbitrary cross-version compatibility; supported version alignment remains the long-term objective.
Checks covered unique names and IPs, reliable resolution, NTP, MTU, no backup/migration/upgrade tasks, no guest lock or pending changes, and bidirectional TCP 22/8006 plus UDP 5405–5412. Ten LAN probes averaged about 1.2 ms with no loss, a small sample rather than a network SLA.
Corosync is sensitive to latency and jitter. Future large disk transfers should not indefinitely compete with its communication path.Proxmox cluster documentation source
Record more than configuration backups
Guest definitions, storage configuration, and job settings were archived and checked. Such backups do not contain guest disks; the DCs had separate existing vzdump archives.
| Baseline | Compare after both operations |
|---|---|
| Configuration SHA-256 | Guest, storage, and job settings |
| Process PID | The same QEMU/LXC processes |
| Uptime | Continuous increase |
| Service probes | DNS, LDAP, SMB, ASR, monitoring |
A running indicator is insufficient: a restarted VM can also appear running. Combined process, uptime, configuration, and service checks support continuity in this particular change.
Create, join, and observe quorum transitions
The following represents the commands with example addresses. Replace documentation addresses and establish verified SSH authorization first:
# Existing workload node
pvecm create lab-pve --link0 address=192.0.2.10
# Completely empty joining node
pvecm add 192.0.2.10 \
--link0 address=192.0.2.20 --use_ssh 1Creation briefly returned Cannot initialize CMAP service. The services subsequently settled into a quorate single-node state. Check final state and logs rather than treating the first status response as definitive:
systemctl status corosync pve-cluster
journalctl -u corosync -u pve-cluster --since '10 minutes ago'
pvecm status
corosync-quorumtool -sDuring joining, membership expanded before the second node started Corosync. The first lost quorum for roughly five seconds, then recovered with both votes. No guests stopped, but this was a real transition. A joining-node failure at that point could leave configuration management read-only.
Acceptance results
Both nodes reported:
Nodes: 2
Expected votes: 2
Total votes: 2
Quorum: 2
Flags: QuorateLink 0 was connected, management services were active, configuration hashes and guest PIDs were unchanged, and uptimes increased. All three DCs reported 0 / 10 failures; ASR, Firecrawl, Prometheus, and macOS SSH probes passed.
The second node could see the cluster resource view, but its local guest directories remained empty. Guest disks had not been copied. These observations validate this operation, not every future change.
Separate configuration visibility from guest continuity
/etc/pve is a cluster configuration view, not shared guest storage. Seeing a VM definition on the second node establishes control-plane visibility; starting or migrating it still requires disks, network, and devices. Identical storage IDs do not prove volumes exist on both hosts.
Archive read-only inventories before and after the change:
pvecm nodes
pvesm status
qm list
pct list
qm pending 101
qm config 101
pvesh get /nodes/pve/qemu/101/status/current --output-format jsonReplace node/VM identifiers; the empty node cannot query a nonexistent local running VM. A configuration hash only covers selected files. A stable host QEMU PID alone cannot exclude an internal guest reboot. Combine guest boot time/uptime, process continuity, and timed application probes. Brief successful DNS/LDAP probes do not prove every client request succeeded.
The approximately five-second quorum transition and guest continuity are different layers. Do not rewrite them as absolutely no impact. If guests remain alive while management becomes read-only, inspect quorum and pmxcfs before restarting guests.
Discuss recovery before creating the cluster
For an incomplete join, inspect both nodes' Corosync/pve-cluster logs, addressing, time, links, and votes. After formation, node removal and configuration cleanup affect the control plane rather than merely undoing a UI operation.
| Scenario | Primary check | Avoid as an unverified shortcut |
|---|---|---|
| One node unavailable, guests still running | Quorum, ownership, outage versus partition | Blind expected-vote changes |
| Network partition, both nodes reachable separately | Possible conflicting resource operations | Forced writes on both sides |
| VM configuration visible, disks absent | Storage and migration dependencies | Assuming disks were replicated |
| Join failure with read-only management | Both logs, membership, recovery path | Overwriting /etc/pve |
Without QDevice or HA, later tests must distinguish third-vote behavior, host failure, partition, storage availability, and business recovery time. A third vote does not restore a lost local disk, and migration success is not failover acceptance. Keep these unperformed exercises explicit.
Two nodes still have a quorum problem
Two votes require both votes for a majority. After one node fails, the other has only 1/2. In this deployment without HA, running guests normally continue, but configuration writes, starts, and migration are constrained by quorum.
An external QDevice should live in an independent failure domain. A VM dependent on the same two hosts is unsuitable as the only independent third vote. QDevice does not supply shared storage, backups, or HA resource configuration. No shutdown or partition drill was performed here.
Placing the second DC on PVE2 improves physical separation; see the DC replacement and IP takeover. It still does not supply a third cluster vote.
The article date is the main KB’s first Git commit date, 2026-09-19 (UTC+8), commit eee5a4c. Experiment dates are stated separately. No production systems were accessed or changed while preparing this article.
