Ceph Storage Cluster with Rook

Guide on how to create a simple Ceph storage cluster with Rook for Kubernetes

This guide walks through deploying a Ceph storage cluster on Talos Linux using Rook, the Kubernetes operator that manages Ceph so you don’t have to run it by hand.

Ceph’s actual storage work is done by OSDs (Object Storage Daemons) — one OSD per disk, each responsible for writing, replicating, and reporting on the data on that disk. Ceph’s total capacity is the sum of its OSDs: mons and the manager keep the cluster coordinated, but no OSDs means no usable storage. Every spare disk mentioned in this guide becomes an OSD.

Prerequisites

Before deploying, make sure your cluster and machines meet the following:

• Talos Linux reserves an entire disk for the OS installation, so you need machines with at least one additional, unused disk for Ceph to use. Only nodes with a spare disk can host an OSD, Rook will not create one on a node without unused storage.
• The block devices or partitions Ceph will use must have no existing partitions or formatted filesystems.
• A supported Kubernetes version. Current Rook (v1.20.x) supports Kubernetes v1.31 through v1.36, check the Rook compatibility matrix for the version you’re installing, since this range shifts with each Rook release.
• Helm v3.
• Enough non-control-plane nodes for your mon count. Rook runs 3 mons by default, one per node, and control-plane nodes are tainted against them, plan for at least 3 worker nodes, or see the mon quorum note under step 5 for smaller clusters.
• Headroom for the CSI driver stack. On top of the mons, the CSI drivers (RBD, CephFS, NFS, NVMe-oF) add real overhead. Since an OSD pod can only run on the node holding its disk, it has nowhere to go if that node is already full.
• Read the Rook Ceph overview before deploying, Ceph is a large system, and the overview explains concepts this guide doesn’t repeat.

Install Rook and create a Ceph cluster

Current Rook installs in three parts: the operator, the Ceph-CSI drivers, and the cluster itself.

1. Add the Rook Helm repository:

   helm repo add rook-release https://charts.rook.io/release
   

2. Install the Rook Operator into the rook-ceph namespace:

   helm install --create-namespace --namespace rook-ceph rook-ceph rook-release/rook-ceph
   

3. Label the rook-ceph namespace so its privileged pods (Ceph daemons need privileged access) aren’t blocked by default Pod Security admission:

   kubectl label namespace rook-ceph pod-security.kubernetes.io/enforce=privileged
   

4. Install the Ceph-CSI drivers chart:

   helm repo add ceph-csi-operator https://ceph.github.io/ceph-csi-operator/
   helm install --create-namespace --namespace rook-ceph ceph-csi-drivers ceph-csi-operator/ceph-csi-drivers
   

5. Install the Ceph cluster chart. The defaults will attempt to use all nodes and all unused disks in the cluster — see the Rook cluster CRD documentation for more specific node, device, and cluster configuration options.

   helm install --create-namespace --namespace rook-ceph rook-ceph-cluster --set operatorNamespace=rook-ceph rook-release/rook-ceph-cluster
   
   If your cluster has fewer non-control-plane nodes than the default mon count (3), cluster creation will stall indefinitely with an event like:

   failed to perform validation before cluster creation: cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false
   

   Relaxing this via helm upgrade --set cephClusterSpec.mon.allowMultiplePerNode=true will fail with a field-ownership conflict, since the operator’s own reconciliation has already taken ownership of several CephCluster spec fields. Patch the resource directly instead:

   kubectl -n rook-ceph patch cephcluster rook-ceph --type merge -p '{"spec":{"mon":{"allowMultiplePerNode":true}}}'
   

   Note that this direct patch means the Helm release’s recorded values will drift from the live CephCluster spec; a later helm upgrade on this release may hit the same conflict again.

Verify the cluster

The Rook operator needs time to bring the Ceph cluster online after the cluster chart is installed. Watch its progress:

kubectl --namespace rook-ceph get cephcluster rook-ceph --watch

The cephcluster resource’s PHASE/MESSAGE fields can lag noticeably behind what the operator is actually doing. For a more real-time view, check the operator’s own logs:

kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=30

Storage classes (ceph-block, ceph-bucket, ceph-filesystem) are created by the Helm chart at install time, independent of cluster health — their presence does not confirm the cluster is actually ready. To check whether OSDs are genuinely up, look for OSD pods directly:

kubectl -n rook-ceph get pods -l app=rook-ceph-osd

Once the cluster reaches HEALTH_OK, the storage classes it creates are ready to use:

kubectl --namespace rook-ceph get cephcluster rook-ceph
kubectl get storageclass

Maintain the cluster during Talos upgrades

By default, Rook configures Ceph with 3 mon instances, so the data in dataDirHostPath can be regenerated from the other mons. Even so, when performing maintenance on a Talos node running Rook (such as a Talos upgrade), take care to keep the Ceph cluster healthy throughout.

Before upgrading a node, confirm the cluster is healthy:

kubectl --namespace rook-ceph get cephclusters.ceph.rook.io rook-ceph

Only proceed if HEALTH reports HEALTH_OK. Upgrade one node at a time, and wait for the cluster to return to HEALTH_OK before touching the next node:

Talos-native

talosctl upgrade --nodes <node-ip> --image ghcr.io/siderolabs/installer:<talos-version>

Omni

Omni manages Talos upgrades itself rather than through a direct talosctl upgrade call:

• For a cluster not managed by a template, open the cluster in the Omni UI and click Update Talos (or the version indicator), select the target version, and click Upgrade.

• For a template-managed cluster, update the talos.version field in the cluster template and sync it:

  omnictl cluster template sync --file cluster-template.yaml
  

Omni performs the upgrade one node at a time and waits for each node to be healthy before continuing, which pairs well with the mon-health caution above.

Wait for the cluster to report HEALTH_OK again before continuing to the next node:

kubectl --namespace rook-ceph wait --timeout=1800s --for=jsonpath='{.status.ceph.health}=HEALTH_OK' cephclusters.ceph.rook.io rook-ceph

Clean up

Cleaning up a Rook Ceph deployment happens in two stages: removing the Ceph cluster itself, then removing the metadata and disk data it left behind on each Talos node.

Remove the Ceph cluster

To remove the ceph cluster:

1. Confirm cluster deletion, then delete the storage classes and Ceph storage types:

   kubectl --namespace rook-ceph patch cephcluster rook-ceph --type merge -p '{"spec":{"cleanupPolicy":{"confirmation":"yes-really-destroy-data"}}}'
   kubectl delete storageclasses ceph-block ceph-bucket ceph-filesystem
   kubectl --namespace rook-ceph delete cephblockpools ceph-blockpool
   kubectl --namespace rook-ceph delete cephobjectstore ceph-objectstore
   kubectl --namespace rook-ceph delete cephfilesystem ceph-filesystem
   

2. Delete the Ceph cluster and its Helm release:

   kubectl --namespace rook-ceph delete cephcluster rook-ceph
   helm --namespace rook-ceph uninstall rook-ceph-cluster
   helm --namespace rook-ceph uninstall ceph-csi-drivers
   

3. If you also want to remove the operator and everything it created, uninstall it, then find and delete the CRDs it left behind. The exact CRD list has changed across Rook versions (the CSI-related ones especially, given the chart restructuring above), so list them from your cluster rather than relying on a hardcoded list:

   helm --namespace rook-ceph uninstall rook-ceph
   kubectl get crds -o name | grep -E 'ceph\.rook\.io|objectbucket\.io'
   

   Review the output, then delete the CRDs it returns.

Remove Rook metadata and wipe the disks

If the Rook Operator was cleanly removed following the steps above, node metadata and disks should already be clean. If cleanup was interrupted, some metadata may remain on the system disk and partition information may remain on the storage disks.

1. Remove leftover node metadata. Replace <storage-node-name> with the node’s name and <dataDirHostPath> with the Rook dataDirHostPath value used when installing the chart (/var/lib/rook if you used the default values). Repeat this for each storage node:

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: disk-clean
   spec:
     restartPolicy: Never
     nodeName: <storage-node-name>
     volumes:
     - name: rook-data-dir
       hostPath:
         path: <dataDirHostPath>
     containers:
     - name: disk-clean
       image: busybox
       securityContext:
         privileged: true
       volumeMounts:
       - name: rook-data-dir
         mountPath: /node/rook-data
       command: ["/bin/sh", "-c", "rm -rf /node/rook-data/*"]
   EOF
   kubectl wait --timeout=900s --for=jsonpath='{.status.phase}=Succeeded' pod disk-clean
   kubectl delete pod disk-clean
   

2. Wipe the partition and filesystem data from each disk used in the Ceph cluster. On Talos, disks are wiped directly using talosctl:

Talos-native

talosctl wipe disk <device> -n <node-ip>

Example:

talosctl wipe disk sdb -n 192.168.1.10

Omni

Download a talosconfig for the cluster, then run the same command with it:

omnictl talosconfig --cluster <cluster-name> ./talosconfig
talosctl --talosconfig ./talosconfig wipe disk <device> -n <node-ip>