Proxmox · Clustering

Proxmox Cluster Setup

A hands-on guide to building a Proxmox VE cluster with pvecm and Corosync, and to the quorum math that decides whether your cluster stays writable when a node drops offline.

Quick idea: A Proxmox cluster ties multiple standalone nodes into one managed unit using Corosync for group communication and quorum voting, and pmxcfs to keep every node’s configuration in sync in real time.
pvecm

The command-line tool used to create, join, inspect, and remove cluster nodes.

Corosync

The cluster engine handling node membership, messaging, and quorum voting between nodes.

pmxcfs

The cluster filesystem that replicates /etc/pve across every node in real time.

What Is a Proxmox Cluster?

A Proxmox VE cluster groups multiple physical servers into a single managed environment. Once nodes are clustered, you manage every VM and container from any node’s web interface, migrate guests between hosts, and layer High Availability on top — none of which a standalone node can offer on its own.

Think of a Proxmox cluster as a group of people who all agree to keep identical copies of the same notebook. Every time one person writes something new — a VM created, a storage pool added — the change gets copied to everyone else’s notebook immediately. If too many people go missing at once, the remaining group refuses to let anyone write anything new until enough of them are back, so no two people can end up disagreeing about what the notebook says.

Under the hood, that notebook is pmxcfs, the Proxmox Cluster Filesystem, which transparently distributes everything under /etc/pve to every node. The “agreeing to keep writing” part is handled by Corosync, which manages node membership and quorum voting, and pvecm is the tool you use to actually create, join, and inspect the cluster.

Important: Joining a node to a cluster overwrites its existing configuration under /etc/pve, and a joining node cannot already hold any guests — their VM/container IDs could conflict with IDs already in use elsewhere in the cluster. Back up any existing guests with vzdump before joining, and restore them under new IDs afterward.

Why Cluster Instead of Standalone Nodes?

A single Proxmox node works fine for a lab box or a single-purpose server, but clustering earns its complexity once you have more than one host to manage.

One Pane of Glass

Manage every node’s VMs and containers from any single node’s web UI — no switching between separate logins.

Live Migration

Move running VMs between nodes with shared storage, for maintenance or load balancing, with no downtime.

High Availability

Automatically restart a guest on another node if the one it was running on fails — HA in Proxmox requires a cluster.

Why Quorum and Corosync Matter

Clustering only helps if the cluster can be trusted to agree on its own state, and that is exactly what Corosync’s quorum mechanism exists to guarantee. Every node gets one vote by default, and the cluster needs a strict majority of votes online before it will accept configuration changes. Lose quorum, and the surviving nodes deliberately switch to a read-only state rather than risk two halves of a split cluster making conflicting changes at the same time.

This matters most right before it becomes an emergency: a HA-managed guest whose node just went dark, a live migration mid-flight when a network link drops, or a config change queued up during a switch reboot. A cluster that lost quorum silently refusing writes looks, from the outside, exactly like a cluster that’s broken — until you check pvecm status.

Practical note: Cluster nodes also need synchronized clocks for Corosync timing and certificate validation to behave. If you haven’t set up time sync on your homelab hosts yet, see the Chrony post before building the cluster, not after.

Requirements Before You Start

Corosync is latency-sensitive, so treat the cluster network as its own concern rather than an afterthought bolted onto whatever NIC was free.

Requirement Detail
Network portsUDP 5405–5412 open between all nodes for Corosync traffic.
LatencyStay under 5 milliseconds between nodes; Corosync is sensitive to jitter, not just raw bandwidth.
Dedicated NICA dedicated 1 Gbit+ physical link is strongly recommended — don’t share it with storage traffic.
SSHTCP 22 reachable between nodes, with root password access when joining.
Time syncClocks synchronized across every node.
Hostname / IPCannot be changed once a node has joined a cluster — get this right before you create or join.

Creating and Joining a Cluster

Run these from the node’s shell. Create the cluster on the first node, then join every additional node to it individually.

# On the first node — create the cluster (name cannot be changed later)
pvecm create homelab-cluster

# Optional: create it with an explicit corosync link address
pvecm create homelab-cluster --link0 10.10.10.1

# Check the local view of cluster status and quorum
pvecm status

# List cluster nodes from the local view
pvecm nodes

# On each additional node — join using the IP of an existing cluster member
pvecm add 10.10.10.1

# Join specifying this node's own local IP on a dedicated corosync link
pvecm add 10.10.10.1 --link0 10.10.10.2

# Remove a node from the cluster (run from a remaining node, after migrating its guests off)
pvecm delnode oldnode
Key rule: pvecm add takes the IP of a node already in the cluster as its main argument — the --link0 value is the joining node’s own local address on that link, not the target’s.

Corosync Configuration

pvecm create and pvecm add generate /etc/pve/corosync.conf for you — you rarely need to hand-edit it, but knowing its shape makes cluster status output much easier to read.

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: uno
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.10.1
  }
  node {
    name: due
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.10.2
  }
  node {
    name: tre
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.10.3
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: homelab-cluster
  config_version: 3
  ip_version: ipv4-6
  secauth: on
  version: 2
}
Section Purpose
nodelistEvery node’s ID, name, vote count, and ring address(es) used for Corosync traffic.
quorumNames the quorum provider — corosync_votequorum is what implements the majority-vote logic.
totemCluster-wide protocol settings: cluster name, encryption, and the config version.
config_versionMust be manually incremented on any hand edit, or the change may not propagate correctly.

Quorum and the Two-Node Problem

A 3-node cluster needs 2 votes for quorum and survives one node failure cleanly. A 2-node cluster needs both votes — meaning either node failing takes the whole cluster read-only, which defeats much of the point of clustering in the first place. Proxmox’s own guidance is to treat three nodes as the practical minimum for a cluster that needs to survive a single failure.

If you’re stuck at two nodes — a common homelab reality — a QDevice gives you a third, external vote without needing a third full Proxmox node.

Setting Up a QDevice

A QDevice runs on a separate, independent machine — a Raspberry Pi or a small VM outside the cluster works fine — and arbitrates quorum over the network rather than participating in the cluster itself.

# On the external QDevice host (not a cluster member)
apt install corosync-qnetd

# On every Proxmox cluster node
apt install corosync-qdevice

# From a cluster node — configure the QDevice (copies the cluster's SSH key automatically)
pvecm qdevice setup 10.10.10.50

# Confirm the QDevice is now counted in cluster status
pvecm status
Production note: Run the QDevice on hardware independent of the cluster it arbitrates for. If it lives on a node that’s part of the cluster, a failure of that one node takes out both a cluster vote and the tie-breaker at the same time.

If you ever need to add or remove a cluster node afterward, remove the QDevice first with pvecm qdevice remove, make the node change, then set the QDevice back up.

pvecm Cheat Sheet

Command What It Does
pvecm create <name>Creates a new cluster on the current node.
pvecm add <ip>Joins the current node to an existing cluster.
pvecm statusShows the local node’s view of cluster health and quorum.
pvecm nodesLists cluster members from the local view.
pvecm delnode <name>Removes a node from the cluster configuration.
pvecm expected <n>Temporarily overrides expected votes to force quorum during an outage.
pvecm qdevice setup <ip>Configures an external QDevice as a tie-breaking vote.
pvecm qdevice removeRemoves a configured QDevice.

Troubleshooting Cheat Sheet

Symptom Likely Cause Fix
pvecm status shows Quorate: No Fewer than a majority of nodes are reachable over the Corosync network. Bring failed nodes or links back online; check pvecm status for which nodes are missing.
Cluster stuck read-only after an outage Not enough nodes came back to reach quorum, and you need to make an emergency change. As a temporary measure only, run pvecm expected 1, then fix the underlying node/network issue.
pvecm add fails with an existing-node error The joining node still has local guests or leftover cluster state from a previous attempt. Back up and remove existing guests with vzdump, then retry the join; use --force only once you understand why the check failed.
Two-node cluster goes read-only whenever either node reboots No third vote exists, so losing either of the two votes breaks the majority. Add a QDevice on independent hardware with pvecm qdevice setup.
Corosync flaky or nodes randomly drop from quorum Cluster traffic is sharing a NIC with heavy storage or VM traffic, causing latency spikes. Move Corosync onto a dedicated NIC or VLAN, and add a second ring/link for redundancy.

Final Thoughts

Most of what makes Proxmox clustering trustworthy is the part that’s invisible day to day — Corosync quietly counting votes and pmxcfs quietly replicating configuration, both handled long before you ever notice them working. The failures that do surface almost always trace back to one of two things: a network that can’t give Corosync the low, stable latency it needs, or a vote count that was never designed to survive the failure you just had.

For a homelab sitting at two nodes, a QDevice is cheap insurance against exactly that second failure mode — it costs a spare Raspberry Pi and ten minutes, against a cluster that goes read-only every time you reboot either box for patching.

Key takeaway: Check pvecm status first whenever a cluster acts up. If it shows Quorate: No, the fix is votes and network reachability, not the guests or storage you’ll otherwise be tempted to blame first.
Next in this series

Once nodes are clustered, shared storage is what actually makes live migration and HA useful rather than theoretical — ZFS and Ceph on Proxmox are natural next topics.