Ceph Storage on Proxmox VE: Architecture, CRUSH Map Rules, and OSD Placement
A comprehensive engineering guide to designing, tuning, and troubleshooting Ceph distributed storage on Proxmox VE—covering CRUSH map hierarchies, custom failure domains, OSD BlueStore topology, and multi-tier pool placement.
size 3 / min_size 2 pool policy, and use CRUSH device classes to keep VM disks off slow rotational media.
Algorithmic pseudo-random data placement calculates object locations deterministically without a central metadata bottleneck.
Direct block device engine bypassing the Linux filesystem layer, backed by RocksDB for lightning-fast metadata and write-ahead logging.
Hierarchical CRUSH rules guarantee that replica copies land across different physical hosts, chassis, or server racks.
What Is Ceph Storage on Proxmox VE?
Ceph is an open-source, massively scalable distributed storage system designed to deliver object, block, and file storage from a unified software cluster. On Proxmox Virtual Environment (PVE), Ceph is tightly integrated directly into the hypervisor stack through the pveceph toolset. This allows administrators to turn local NVMe, SSD, or SATA disks installed across hypervisor hosts into a resilient, shared storage pool without buying dedicated SAN appliances or proprietary hardware controllers.
Think of traditional shared storage—like a central NAS or dual-controller SAN—as a traditional postal warehouse where a single dispatch clerk sits behind a desk with a massive paper logbook. Every time a package arrives or needs to be retrieved, workers must queue up at the front desk and ask the clerk to look up the exact aisle, shelf, and bin number in the master ledger. If the ledger gets corrupted, or if the clerk gets overwhelmed by thousands of simultaneous requests, the entire warehouse grinds to a halt.
Ceph with CRUSH operates on an entirely different philosophy. Think of it like a smart automated distribution facility where every delivery driver carries a pocket calculator, a floor plan of the facility, and three simple rules: store three copies of every parcel, never put two copies in the same aisle, and store fragile items on lower tiers. When a parcel arrives, the driver simply runs the parcel’s barcode through a mathematical formula. The math immediately yields the exact shelf and bin coordinates. There is no front desk, no central logbook, and no single bottleneck. Every node calculates data placement independently, instantly, and deterministically.
Inside a Proxmox VE Ceph cluster, several specialized daemons work in concert:
- Ceph Monitor (ceph-mon): Maintains the master cluster map, tracking the health and active status of all nodes, OSDs, and pools using the Paxos consensus algorithm. Monitors require an odd quorum (3 or 5 nodes) to operate safely.
- Ceph Manager (ceph-mgr): Runs alongside the monitors to handle operational metrics, orchestrate PG autoscaling, expose telemetry, and serve the Ceph dashboard and Prometheus exporters.
- Object Storage Daemon (ceph-osd): The engine that manages an individual physical storage disk. Each OSD handles reading, writing, replication, rebalancing, and heartbeat monitoring for the data chunks assigned to it.
- Metadata Server (ceph-mds): Manages metadata hierarchies for CephFS shared POSIX filesystems (optional if you only run RBD block storage for VM virtual disks).
Why Ceph Instead of Shared ZFS or Traditional SAN/NAS?
Engineers coming from a traditional homelab or mid-market enterprise background frequently ask why they should absorb the complexity of Ceph when local ZFS storage pools on Proxmox already offer snapshotting, checksumming, and exceptional NVMe write speeds.
The answer comes down to one operational requirement: shared state without a single point of failure.
Blazing fast NVMe/SATA read-write performance, but virtual disks are trapped on a single physical host. If the motherboard dies, the VM stays offline until the host is restored or replicated storage catches up.
Shared NFS or iSCSI storage allows instant VM migration, but the SAN itself represents a single failure domain, power hog, and significant financial investment.
Combines local disk pools across all hypervisor nodes into a single shared distributed namespace. Live migration is instant, storage scales horizontally, and any single node can catch fire without downtime.
With local ZFS, running a high-availability Proxmox cluster requires scheduled asynchronous replication (e.g., ZFS send/receive every 5 to 15 minutes). If a node fails suddenly, you must choose between waiting for replication lag to clear or accepting up to 15 minutes of transactional data loss.
With Ceph RADOS Block Device (RBD), storage is intrinsically shared across all participating Proxmox nodes. When a hypervisor node suffers an abrupt hardware failure, Proxmox HA detects the loss and immediately boots the displaced virtual machines on surviving nodes in seconds, with zero data loss, because the VM’s storage was never tethered to the dead physical machine.
Why Proxmox High Availability and Clustering Depend on Ceph
A stable Proxmox High Availability (HA) cluster requires two tightly coupled layers: a low-latency quorum voting mechanism managed by Corosync, and a resilient shared storage backplane.
When you run VM workloads on Ceph RBD, live migration does not copy a single byte of disk data across the wire. Proxmox simply pauses the guest vCPUs, copies the dirty RAM pages to the target node over the management network, switches the RBD lock, and resumes execution. For heavy SQL databases and Active Directory Domain Controllers, this transforms multi-gigabyte disk migrations from a 10-minute network drag into a seamless 2-second handoff.
However, this interdependence introduces a critical enterprise failure mode that catches many administrators off-guard: the Corosync starvation death spiral.
To prevent this, Ceph architecture demands strict physical or logical network separation between cluster management, client storage access, and internal OSD replication.
Required Ports, Protocols & Firewall Rules
Ceph daemons communicate across two distinct logical networks: the Public Network (where Proxmox hypervisor clients and VMs query monitors and read/write OSD data) and the Cluster Network (where OSD daemons exchange peer heartbeats, monitor data consistency, and stream replication/backfill data).
When configuring the Proxmox VE firewall or upstream switch access control lists (ACLs), ensure the following port matrix is explicitly permitted:
| Port / Range | Protocol | Direction | Source / Destination | Purpose & Service |
|---|---|---|---|---|
TCP 3300 |
TCP | Inbound / Outbound | Proxmox Nodes ↔ Monitors | Ceph Messenger v2 (msgr2) modern secure wire protocol. |
TCP 6789 |
TCP | Inbound / Outbound | Proxmox Nodes ↔ Monitors | Ceph Messenger v1 (msgr1) legacy monitor communication. |
TCP 6800–7300 |
TCP | Inbound / Outbound | OSD Nodes ↔ OSD Nodes / Clients | Dynamic ports for OSD daemons, Manager (ceph-mgr), and Metadata Servers (ceph-mds). |
TCP 8443 |
TCP | Inbound | Admin Workstations → PVE Nodes | Ceph Manager Web Dashboard (HTTPS management interface). |
TCP 9283 |
TCP | Inbound | Prometheus Scraper → PVE Nodes | Ceph Manager built-in Prometheus metrics exporter. |
UDP 5405–5412 |
UDP | Bidirectional | Proxmox Cluster Nodes | Corosync cluster engine heartbeats and quorum synchronization. |
TCP 8006 |
TCP | Inbound | Admin Workstations → PVE Nodes | Proxmox VE Web Management Console and REST API. |
TCP 22 |
TCP | Bidirectional | Cluster Nodes ↔ Cluster Nodes | SSH node communication, tunnel setup, and live migration orchestration. |
MTU 9000) across your dedicated Ceph storage interfaces and physical switches. Ceph transfers large 4MB object chunks during VM disk operations and OSD backfilling; raising the MTU significantly reduces CPU interrupt overhead and eliminates Ethernet packet fragmentation.
Ceph Architecture Deep Dive: OSDs, BlueStore, and RocksDB
In modern releases of Ceph on Proxmox VE (including Ceph Reef and Squid), the default and only supported OSD storage engine is BlueStore.
In older versions of Ceph (Jewel and earlier), the storage engine was FileStore, which wrote Ceph objects as standard files onto a conventional Linux filesystem (like XFS or ext4). This introduced double-journaling penalties: data had to be written first to a POSIX journal and then committed to the filesystem, severely choking write IOPS.
BlueStore eliminates the Linux filesystem entirely. It treats the physical disk as a raw block device, consuming bare disk sectors directly:
+--------------------------------------------------------------------------+
| Ceph OSD Daemon (ceph-osd) |
+--------------------------------------------------------------------------+
| BlueStore |
+------------------------------------+-------------------------------------+
| BlueFS (Internal Meta) | |
+------------------------------------+ Raw Block Device |
| RocksDB WAL | RocksDB Data | (Data Payloads) |
| (Fast NVMe) | (Fast NVMe) | (NVMe / SSD / HDD) |
+----------------+-------------------+-------------------------------------+
Inside an individual BlueStore OSD, storage is split into three functional spaces:
- Primary Block Device: Holds the raw object data payloads. In an all-flash cluster, this is your NVMe or enterprise SATA SSD. In hybrid nodes, this is typically a high-capacity rotational HDD.
- RocksDB Write-Ahead Log (WAL): An append-only log that commits small metadata updates and transaction records instantly, ensuring crash consistency before changes are committed to the database.
- RocksDB Database (DB): Stores object metadata, checksums, allocation maps, and internal object namespaces.
When building hybrid Ceph nodes (combining spinning rotational disks for capacity with fast flash drives for acceleration), BlueStore allows you to place the RocksDB and WAL partitions on a blazing-fast enterprise NVMe drive while keeping the primary block data on the slow HDD:
The CRUSH Map: Hierarchy, Buckets, and Failure Domains
At the heart of Ceph’s zero-bottleneck architecture lies CRUSH (Controlled Replication Under Scalable Hashing).
When a virtual machine writes a block to a Proxmox virtual disk, Ceph stripes that block into discrete 4MB objects. Each object is assigned a unique name (such as rbd_data.104a32.000000000004). Placement does not happen by random choice or central decree. Instead, the object name is run through an immutable mathematical pipeline:
[VM Write: 4MB Object]
│
hash(object_id) → Deterministic 32-bit Integer
│
Integer % Total_PGs → Target Placement Group (e.g. PG 2.1a)
│
CRUSH(PG 2.1a, Cluster_Map, Rule) → Ordered OSD List: [osd.1, osd.5, osd.8]
│
[Client streams payload directly to Primary OSD (osd.1)]
│
[osd.1 replicates payload synchronously to Secondary OSDs (osd.5, osd.8)]
│
[osd.1 acknowledges write to VM once min_size replicas confirm]
CRUSH relies on a hierarchical topology map that mirrors the physical layout of your datacenter or homelab. This hierarchy is built from nested containers called buckets:
root: The top of the tree (defaults todefault).datacenter/room: Logical site definitions for multi-building deployments.rack: Server rack cabinets sharing common power distribution units (PDUs) and top-of-rack switches.host: An individual physical Proxmox hypervisor server.osd: The leaf nodes of the tree, representing individual physical disks.
The power of CRUSH lies in configuring the failure domain. By setting the failure domain to type host (the default in Proxmox VE), CRUSH enforces that no two replicas of the same placement group can ever be placed on OSDs that reside within the same physical server. Even if an entire server loses power, catches fire, or blows a power supply, the remaining two copies survive untouched on other nodes.
In multi-rack enterprise environments, you can elevate the failure domain to type rack. CRUSH will then guarantee that even if an entire top-of-rack switch fails or a rack PDU trips, every single object maintains active surviving replicas in completely separate server cabinets.
Designing Custom CRUSH Rules & OSD Device Classes
In a real-world infrastructure, you rarely have identical disks across every machine. A typical Proxmox cluster might feature high-speed enterprise NVMe drives for mission-critical database workloads, high-end SATA SSDs for general-purpose application servers, and large rotational HDDs for ISO images, template storage, and cold backup archives.
If you throw all these disks into a single default Ceph pool, CRUSH will blindly stripe objects across both NVMes and HDDs. Your fast database queries will frequently hang waiting for an 8TB spinning disk to seek.
To solve this, Ceph introduces Device Classes. When an OSD starts up, Ceph automatically probes the underlying hardware and tags the OSD with a class: nvme, ssd, or hdd.
You can inspect your cluster’s device classes directly from the Proxmox CLI:
# List all detected device classes in the cluster
ceph osd crush class ls
# View OSDs grouped by their assigned device class
ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 5.82089 root default
-3 1.94030 host pve-01
0 nvme 0.97015 osd.0 up 1.00000 1.00000
1 hdd 0.97015 osd.1 up 1.00000 1.00000
-5 1.94030 host pve-02
2 nvme 0.97015 osd.2 up 1.00000 1.00000
3 hdd 0.97015 osd.3 up 1.00000 1.00000
-7 1.94030 host pve-03
4 nvme 0.97015 osd.4 up 1.00000 1.00000
5 hdd 0.97015 osd.5 up 1.00000 1.00000
To create segregated storage tiers, you construct custom CRUSH rules that filter by device class. For example, to build a lightning-fast pool restricted exclusively to NVMe drives with host-level redundancy:
# Create a replicated CRUSH rule named 'rule-nvme' targeting NVMe disks with host failure domain
ceph osd crush rule create-replicated rule-nvme default host nvme
# Create a second rule named 'rule-hdd' targeting rotational disks for bulk storage
ceph osd crush rule create-replicated rule-hdd default host hdd
Once the rules are established, you can assign them to specific storage pools:
# Create a high-performance VM pool bound to the NVMe CRUSH rule
pveceph pool create vm-nvme-pool --crush-rule rule-nvme --pg_num 128
# Create a bulk storage pool bound to the HDD CRUSH rule
pveceph pool create bulk-hdd-pool --crush-rule rule-hdd --pg_num 64
For complex multi-tier topologies (such as segregating hosts by rack or datacenter floor), you can decompile the binary CRUSH map, edit the bucket tree and rule steps directly in a text editor, and recompile it back to the cluster:
# Step 1: Export the current live CRUSH map as a binary file
ceph osd getcrushmap -o /tmp/crush.bin
# Step 2: Decompile the binary map into human-readable text
crushtool -d /tmp/crush.bin -o /tmp/crush.txt
# Step 3: Inspect and edit the text map
nano /tmp/crush.txt
# Step 4: Recompile the text map into binary format
crushtool -c /tmp/crush.txt -o /tmp/crush.bin.new
# Step 5: Safely inject the new CRUSH map into the active cluster
ceph osd setcrushmap -i /tmp/crush.bin.new
Inside /tmp/crush.txt, a custom CRUSH rule looks like this:
rule rule-nvme {
id 1
type replicated
step take default class nvme
step chooseleaf firstn 0 type host
step emit
}
The step take default class nvme instruction tells CRUSH to only consider OSDs marked with the nvme class under the default root bucket. The step chooseleaf firstn 0 type host instruction tells CRUSH to select leaf OSDs such that each chosen disk lives inside a distinct host bucket. Finally, step emit flushes the resulting OSD list back to the client.
Placement Groups (PGs) and the PG Autoscaler
A common point of confusion for new Ceph operators is the relationship between objects, Placement Groups (PGs), and OSDs.
If Ceph tracked the physical location of millions of individual 4MB objects directly, the cluster map would consume gigabytes of RAM on every node and monitors would melt down processing updates. Instead, Ceph groups millions of objects into a few hundred or thousand Placement Groups. A Placement Group is simply a virtual collection bucket. Ceph manages replication, peering, and scrubbing at the PG level, not the object level.
The standard engineering rule of thumb for replicated pools (size 3) is to target approximately 100 Placement Groups per OSD across the entire cluster.
- Too few PGs: Data distributes unevenly. Some OSDs fill to 90% capacity while others sit idle at 30%, triggering premature cluster-full alerts.
- Too many PGs: Each OSD must maintain peering states, heartbeats, and scrub logs for thousands of groups, driving up CPU usage and exhausting system RAM.
Proxmox VE includes the native Ceph PG Autoscaler, which monitors pool utilization and automatically scales PG counts up or down to match actual data footprint. You can review the autoscaler’s recommendations at any time:
# Check PG autoscaling status and recommendations across all pools
ceph osd pool autoscale-status
POOL SIZE TARGET SIZE RATE RAW CAPACITY %RAM RATIO TARGET RATIO PG_NUM NEW PG_NUM AUTOSCALE
device_health_metrics 0 3.0 5.82Ti 0 0.0000 1 on
vm-nvme-pool 420Gi 3.0 5.82Ti 7 0.2114 0.7000 128 on
bulk-hdd-pool 180Gi 3.0 5.82Ti 3 0.0906 0.3000 64 on
To prevent the autoscaler from wildly adjusting PG counts during initial cluster setup when pools are mostly empty, you can guide it by declaring your expected pool capacity using the target_size_ratio parameter:
# Instruct the autoscaler that vm-nvme-pool will eventually consume 70% of available NVMe storage
ceph osd pool set vm-nvme-pool target_size_ratio 0.70
# Instruct the autoscaler that bulk-hdd-pool will consume 30% of available HDD storage
ceph osd pool set bulk-hdd-pool target_size_ratio 0.30
Configuration & Hands-On CLI Management
In Proxmox VE, Ceph configuration is stored in a shared cluster filesystem file: /etc/pve/ceph.conf. Because this file lives under /etc/pve/, changes made on any single node are automatically synchronized across all cluster members via pmxcfs.
Here is a production-hardened /etc/pve/ceph.conf example for a 3-node enterprise lab cluster:
# /etc/pve/ceph.conf
# Global cluster configuration for Proxmox VE Ceph cluster
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 10.10.20.0/24
public_network = 10.10.10.0/24
fsid = 4d28e7e2-1b1f-4d98-8c19-9c59239a0ef1
mon_allow_pool_delete = false
mon_host = 10.10.10.11 10.10.10.12 10.10.10.13
ms_bind_msgr2 = true
osd_pool_default_min_size = 2
osd_pool_default_size = 3
[client]
rbd_cache = true
rbd_cache_size = 67108864
rbd_cache_max_dirty = 50331648
rbd_cache_target_dirty = 33554432
[osd]
bluestore_cache_autotune = true
osd_memory_target_autotune = true
osd_deep_scrub_interval = 604800
osd_scrub_begin_hour = 1
osd_scrub_end_hour = 6
Let’s break down the critical architectural directives in this configuration:
| Directive | Value | Architectural Purpose |
|---|---|---|
cluster_network |
10.10.20.0/24 |
Defines the isolated subnet dedicated to internal OSD replication, recovery, and heartbeats. |
public_network |
10.10.10.0/24 |
Defines the client-facing subnet where hypervisors, monitors, and VMs connect. |
mon_allow_pool_delete |
false |
Prevents catastrophic accidental deletion of active pools from the CLI or Web UI. |
osd_pool_default_size |
3 |
Sets the standard replica count so every piece of data exists across three independent nodes. |
osd_pool_default_min_size |
2 |
Enforces that at least two active replicas must confirm a write before acknowledging success to the client. |
osd_deep_scrub_interval |
604800 |
Runs bit-rot consistency deep scrubbing once every 7 days (604,800 seconds) rather than daily. |
osd_scrub_begin/end_hour |
1 / 6 |
Restricts disk scrubbing operations to off-peak hours (between 01:00 and 06:00 local time). |
Here is the ordered CLI playbook for initialising Ceph, deploying monitors, adding OSDs, and managing the disk lifecycle:
# ==============================================================================
# PHASE 1: INITIALISE CEPH ON THE FIRST PROXMOX NODE
# ==============================================================================
# Install Ceph packages on the host
pveceph install
# Initialise Ceph with explicit public and cluster replication networks
pveceph init --network 10.10.10.0/24 --cluster-network 10.10.20.0/24
# Create the primary Monitor and Manager daemons on the first node
pveceph mon create
pveceph mgr create
# Repeat the mon and mgr create commands on nodes 2 and 3 to establish Paxos quorum:
# ssh pve-02 "pveceph mon create && pveceph mgr create"
# ssh pve-03 "pveceph mon create && pveceph mgr create"
# ==============================================================================
# PHASE 2: DEPLOYING BLUESTORE OSDS
# ==============================================================================
# Create an all-flash OSD using a single NVMe disk
pveceph osd create /dev/nvme0n1
# Create a hybrid OSD using a slow rotational HDD with RocksDB/WAL on fast NVMe
pveceph osd create /dev/sdb --db_dev /dev/nvme0n1 --db_size 60
# Verify active OSD status, storage footprint, and tree distribution
ceph osd status
ceph osd df
ceph osd tree
# ==============================================================================
# PHASE 3: SAFE OSD REPLACEMENT & DECOMMISSIONING WORKFLOW
# ==============================================================================
# Step 1: Mark the failing OSD 'out' so Ceph begins migrating data off it
ceph osd out 3
# Step 2: Monitor cluster health until all degraded PGs rebalance to 'active+clean'
ceph status
# Step 3: Stop the daemon process on the host housing the OSD
systemctl stop ceph-osd@3
# Step 4: Safely unmap and purge the OSD from the CRUSH map and authentication keys
ceph osd purge 3 --yes-i-really-mean-it
# Step 5: Physically swap the drive and provision the replacement OSD
pveceph osd create /dev/sdb
Production Gotchas, Failure Modes & Cheat Sheet
Operating Ceph at scale reveals failure modes that are never mentioned in glossy architecture brochures. Here are the four most dangerous traps to prepare for:
size 2 and min_size 1 to conserve disk space. This is playing Russian roulette with your data. If one disk fails, Ceph drops to a single copy. If that single surviving disk hits a single bad sector (an Unrecoverable Read Error) while trying to rebuild the replacement disk, the entire placement group enters an unrecoverable incomplete or down state, permanently corrupting your virtual disk. In production, always enforce size 3 / min_size 2.
nearfull). The moment any single OSD reaches 90% (full), Ceph immediately locks down all write operations across the entire cluster to prevent data corruption. VMs immediately freeze with disk I/O errors. Because CRUSH distributes data unevenly by a few percentage points, one disk can hit 90% while the cluster average is only 78%. Monitor per-OSD utilization with ceph osd df and rebalance early using ceph osd reweight-by-utilization.
Use this quick reference to interpret cluster health states and operational indicators:
| Status Indicator | Operational Meaning | Required Action |
|---|---|---|
HEALTH_OK |
All monitors are in quorum, all OSDs are up and in, and all PGs are active+clean. | Cluster operating normally; no administrative intervention required. |
HEALTH_WARN: degraded data redundancy |
One or more OSDs are down or out; some objects have fewer than 3 replicas. | Identify failing disk with ceph health detail; verify backfill progress. |
HEALTH_WARN: mons pve-03 are down |
A monitor daemon is offline; quorum is degraded but functional if >50% survive. | Check node reachability and restart daemon: systemctl restart ceph-mon@pve-03. |
HEALTH_ERR: 1 nearfull osd(s) |
An individual OSD has exceeded 85% raw disk capacity. | Add capacity or reweight OSDs using ceph osd reweight-by-utilization. |
HEALTH_ERR: full osds |
An OSD has crossed the 90% hard threshold; client write operations are frozen. | Immediately expand OSD capacity or temporarily raise threshold to restore write access. |
PG state: active+undersized |
The placement group is actively serving I/O, but lacks its full complement of replicas. | Normal during OSD replacement or node reboot; clears once backfilling completes. |
When an incident strikes, use this break-fix diagnostic cheat sheet:
| Symptom | Likely Root Cause | Verified Resolution |
|---|---|---|
| VM disk I/O locks up; writes hang indefinitely | Cluster has crossed the full_ratio (90%) ceiling on at least one OSD. |
Temporarily raise the full threshold to restore emergency write I/O: ceph osd set-full-ratio 0.95, then purge unused snapshots or add new disks. |
| Proxmox nodes randomly reboot during OSD rebalancing | Ceph recovery traffic is saturating the link shared with Corosync, dropping cluster heartbeats. | Throttle backfill bandwidth immediately: ceph tell 'osd.*' injectargs '--osd_max_backfills=1 --osd_recovery_max_active=1', then separate the networks. |
| High I/O latency on hybrid HDD OSDs | RocksDB has exhausted its dedicated NVMe partition and spilled over onto slow rotational media. | Inspect BlueStore DB space allocation with ceph daemon osd.X perf dump and redeploy the OSD with a larger --db_size partition. |
Placement groups stuck in peering state |
Network partition or firewall rule blocking OSD-to-OSD communication on ports 6800–7300. | Verify OSD port reachability across all nodes using nc -zv <osd-ip> 6800-7300 and inspect local iptables rules. |
| Monitor election fails to form quorum | Clock skew between Proxmox nodes exceeds the allowed 50ms drift threshold. | Verify NTP synchronization across all hosts: chronyc tracking and synchronize system clocks before restarting monitors. |
Automated Cluster Health Validation
In an enterprise or advanced homelab environment, you should never wait for an engineer to log into the Proxmox GUI to discover an unhealthy Ceph cluster.
Below is a concise, production-ready Bash monitoring script that validates Ceph health, OSD states, degraded placement groups, and disk fill ratios. It exits with code 0 on healthy state and code 1 on any actionable degradation, making it ideal for integration into cron, Prometheus textfile collectors, or synthetic CI pipelines:
#!/usr/bin/env bash
# ==============================================================================
# verify-ceph-health.sh — Automated Ceph Cluster Health Gate
# Run on any Proxmox VE cluster member to validate storage stability
# ==============================================================================
set -euo pipefail
echo "=== [1/4] Checking Ceph Overall Health Status ==="
HEALTH_STATUS=$(ceph health --format json | grep -o '"status":"[^"]*' | cut -d'"' -f4)
echo "Current Cluster Health: ${HEALTH_STATUS}"
if [[ "${HEALTH_STATUS}" == "HEALTH_ERR" ]]; then
echo "[FAIL] Critical Ceph health error detected!"
ceph health detail
exit 1
fi
echo "=== [2/4] Verifying OSD Daemon Availability ==="
TOTAL_OSD=$(ceph osd stat --format json | grep -o '"num_osds":[0-9]*' | cut -d':' -f2)
UP_OSD=$(ceph osd stat --format json | grep -o '"num_up_osds":[0-9]*' | cut -d':' -f2)
IN_OSD=$(ceph osd stat --format json | grep -o '"num_in_osds":[0-9]*' | cut -d':' -f2)
echo "OSDs: Total=${TOTAL_OSD}, Up=${UP_OSD}, In=${IN_OSD}"
if [[ "${UP_OSD}" -ne "${TOTAL_OSD}" ]]; then
echo "[WARN] One or more OSDs are down!"
ceph osd tree | grep -E "down|out"
exit 1
fi
echo "=== [3/4] Inspecting Placement Group States ==="
DEGRADED_PGS=$(ceph pg stat --format json | grep -o '"num_pg_by_state":\[[^]]*\]' || true)
if echo "${DEGRADED_PGS}" | grep -qE "incomplete|stale|down"; then
echo "[FAIL] Corrupted or unreachable placement groups found!"
ceph pg dump_stuck stale
ceph pg dump_stuck inactive
exit 1
fi
echo "[PASS] All Placement Groups are active and serving I/O."
echo "=== [4/4] Auditing OSD Disk Capacity Limits ==="
MAX_UTIL=$(ceph osd df --format json | grep -o '"utilization":[0-9.]*' | cut -d':' -f2 | sort -nr | head -n1)
echo "Highest OSD Disk Utilization: ${MAX_UTIL}%"
if (( $(echo "${MAX_UTIL} > 85.0" | bc -l) )); then
echo "[WARN] OSD utilization exceeds 85% nearfull threshold!"
ceph osd df | sort -k 7 -nr | head -n 5
exit 1
fi
echo "=============================================================================="
echo "[SUCCESS] Ceph cluster is fully healthy, balanced, and ready for production."
echo "=============================================================================="
exit 0
Final Thoughts
Ceph on Proxmox VE is one of the most powerful storage architectures available to systems engineers. It completely demystifies enterprise storage, replacing proprietary vendor lock-in and million-dollar SAN controllers with commodity hardware, mathematical hashing, and true horizontal scalability.
However, Ceph’s resilience is only as robust as the architecture underneath it. If you treat Ceph like a simple software RAID array, cram it onto a single shared 1GbE network link, or compromise on replication rules, it will ruthlessly expose those shortcuts the moment a drive or node falters.
Design your cluster with dedicated 10GbE or 25GbE storage networks, enforce strict size 3 / min_size 2 pool policies, structure your CRUSH maps to reflect your true physical failure domains, and monitor RocksDB utilization. When built correctly, you will have a rock-solid storage backplane capable of surviving motherboard meltdowns, drive failures, and hypervisor upgrades without a single second of VM downtime.
Next, we will explore pfSense site-to-site WireGuard VPN architectures, detailing how to interconnect multi-site enterprise labs with routed subnets, redundant gateways, and MTU-optimized transport tunnels.
Discover more from Karanth.ovh
Subscribe to get the latest posts sent to your email.