Alauda Container Platform

Block Storage Disaster Recovery

RBD Mirror is a feature of Ceph Block Storage (RBD) that enables asynchronous data replication between different Ceph clusters, providing cross-cluster Disaster Recovery (DR). Its core function is to synchronize data in a primary-backup mode, ensuring rapid service takeover by the backup cluster when the primary cluster fails.

WARNING
  • RBD Mirror performs incremental synchronization based on snapshots, with a default snapshot interval of once per hour (configurable). The differential data between primary and backup clusters typically corresponds to writes within one snapshot cycle.
  • RBD Mirror only provides underlying storage data backup and does not handle Kubernetes resource backups. Please use the platform's Backup and Restore feature to back up PVC and PV resources.

TOC

Terminology

TermExplanation
Primary ClusterThe cluster currently providing storage services.
Secondary ClusterThe standby cluster used for backup purposes.

Backup Configuration

Prerequisites

  • Prepare two clusters capable of deploying Alauda Build of Rook-Ceph: a Primary cluster and a Secondary cluster, with network connectivity between them.
  • Both clusters must run the same platform version (v3.12 or later).
  • Create distributed storage services in both Primary and Secondary clusters.
  • Create block storage pools with identical names in both Primary and Secondary clusters.
  • Please ensure that the following two images have been uploaded to the platform's private image repository:
    • quay.io/csiaddons/k8s-controller:v0.12.0 -> <registry>/csiaddons/k8s-controller:v0.12.0
    • quay.io/csiaddons/k8s-sidecar:v0.12.0 -> <registry>/csiaddons/k8s-sidecar:v0.12.0

Procedures

Bootstrap Peers(Primary <-> Secondary)

  1. Enable Mirroring for Primary Cluster's Block Storage Pool

    Execute the following command on both Primary and Secondary clusters' Control nodes:

    Command
    Output
    kubectl -n rook-ceph patch cephblockpool <block-pool-name> \
  --type merge -p '{"spec":{"mirroring":{"enabled":true,"mode":"image"}}}'

    Parameters:

    • <block-pool-name>: Block storage pool name.

  2. Obtained Peer Token

    This token serves as the critical credential for establishing mirror connections between clusters.

    Execute the following command on both Primary and Secondary clusters' Control nodes:

    Command
    Output
    kubectl get secret -n rook-ceph \
  $(kubectl get cephblockpool.ceph.rook.io <block-pool-name> -n rook-ceph -o jsonpath='{.status.info.rbdMirrorBootstrapPeerSecretName}') \
  -o jsonpath='{.data.token}' | base64 -d

  3. Create Peer Token Secret in Peer Cluster

    Execute the following command on both Primary and Secondary cluster's Control node:

    Command
    Output
    kubectl -n rook-ceph create secret generic rbd-peer-site-secret \
  --from-literal=token=<token> \
  --from-literal=pool=<block-pool-name>

    Parameters:

    • <token>: Token obtained from Step 2.

      On the Primary cluster, configure this field using the token obtained from the Secondary cluster.

      On the Secondary cluster, configure this field using the token obtained from the Primary cluster.

    • <block-pool-name>: Block storage pool name.

  4. Patch Peer Secret for Block Storage Pool

    Execute the following command on both Primary and Secondary cluster's Control node:

    Command
    Output
    kubectl -n rook-ceph patch cephblockpool <block-pool-name> --type merge -p \
'{
  "spec": {
    "mirroring": {
      "peers": {
        "secretNames": [
          "rbd-peer-site-secret"
        ]
      }
    }
  }
}'

    Parameters:

    • <block-pool-name>: Block storage pool name.

  5. Deploy Mirror Daemon

    This daemon is responsible for monitoring and managing RBD mirror synchronization processes, including data synchronization and error handling.

    Execute the following command on both Primary and Secondary cluster's Control node:

    Command
    Output
    cat << EOF | kubectl apply -f -
apiVersion: ceph.rook.io/v1
kind: CephRBDMirror
metadata:
  name: rbd-mirror
  namespace: rook-ceph
spec:
  count: 1
EOF

  6. Verify Mirror Status

    Execute the following command on both Primary and Secondary cluster's Control node:

    Command
    Output
    kubectl get cephblockpools.ceph.rook.io <block-pool-name> -n rook-ceph -o jsonpath='{.status.mirroringStatus.summary}'

    Parameters:

    • <block-pool-name>: Block storage pool name.

Setup Environment For Volume Replication

This feature enables efficient data replication and synchronization without interrupting primary application operations, enhancing system reliability and availability.

  1. Setup CsiAddons Controller

    Execute the following commands on both Primary and Secondary clusters' Control nodes:

    kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.12.0/deploy/controller/crds.yaml
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.12.0/deploy/controller/setup-controller.yaml
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.12.0/deploy/controller/rbac.yaml
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.12.0/deploy/controller/csi-addons-config.yaml

kubectl -n csi-addons-system set image deployment/csi-addons-controller-manager manager=<registry>/csiaddons/k8s-controller:v0.12.0

    Parameters:

    • <registry>: Registry address of platform.

  2. Enable CsiAddons sidecar

    Execute the following commands on both Primary and Secondary clusters' Control nodes:

    kubectl patch cm rook-ceph-operator-config -n rook-ceph --type json --patch \
'[
  {
    "op": "add",
    "path": "/data/CSI_ENABLE_OMAP_GENERATOR",
    "value": "true"
  },
  {
    "op": "add",
    "path": "/data/CSI_ENABLE_CSIADDONS",
    "value": "true"
  },
  {
    "op": "add",
    "path": "/data/ROOK_CSIADDONS_IMAGE",
    "value": "<registry>/csiaddons/k8s-sidecar:v0.12.0"
  }
]'

    Wait for all csi pods to restart successfully

    kubectl get po -n rook-ceph -w | grep csi

  3. Create VolumeReplicationClass

    Execute the following commands on both Primary and Secondary clusters' Control nodes:

    Command
    Output
    cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplicationClass
metadata:
  name: rbd-volumereplicationclass
spec:
  provisioner: rook-ceph.rbd.csi.ceph.com
  parameters:
    mirroringMode: snapshot
    schedulingInterval: "<scheduling-interval>"
    replication.storage.openshift.io/replication-secret-name: rook-csi-rbd-provisioner
    replication.storage.openshift.io/replication-secret-namespace: rook-ceph
EOF
    1. <scheduling-interval>: Scheduling interval, (e.g., schedulingInterval: "1h" indicates execution every 1 hour.)

Enable Mirror for PVC

Execute the following command on the Primary cluster's Control node:

Command
Output
cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  volumeReplicationClass: rbd-volumereplicationclass
  replicationState: primary
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <pvc-name>
EOF
  1. <vr-name>: The name of the VolumeReplication object, recommended to be the same as the PVC name.
  2. <namespace>: The namespace to which the VolumeReplication belongs, which must be the same as the PVC namespace.
  3. <pvc-name>: The name of the PVC for which Mirror needs to be enabled.

Note After enabling, the RBD image in the Secondary cluster becomes read-only.

Planned Migration

Use cases: Datacenter maintenance, technology refresh, disaster avoidance, etc.

Relocation

The Relocation operation is the process of switching production to a backup facility(normally your recovery site) or vice versa.

For relocation, access to the image on the primary site should be stopped. The image should now be made primary on the secondary cluster so that the access can be resumed there.

Prerequisites

  • The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

Procedures

Follow the below steps for planned migration of workload from the Primary cluster to the Secondary cluster:

  1. Scale down all the application pods which are using the mirrored PVC on the Primary cluster.

  2. Update VolumeReplications for all the PVCs which mirroring is enabled on the Primary cluster.

    Set spec.replicationState to secondary.

  3. Create VolumeReplications for all the PVCs for which mirroring is enabled on the Secondary.

    Example
    cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  volumeReplicationClass: rbd-volumereplicationclass
  replicationState: primary
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <pvc-name>
EOF
    1. <vr-name>: The name of the VolumeReplication object, recommended to be the same as the PVC name.
    2. <namespace>: The namespace to which the VolumeReplication belongs, which must be the same as the PVC namespace.
    3. <pvc-name>: The name of the PVC for which Mirror needs to be enabled.

  4. Check VolumeReplication CR status to verify if the image is marked primary on the secondary site.

  5. Once the Image is marked as primary, the PVC is now ready to be used. Now, we can scale up the applications to use the PVC.

Disaster Recovery

Use cases: Natural disasters, Power failures, System failures, and crashes, etc.

Failover (abrupt shutdown)

In case of Disaster recovery, create VolumeReplication CR at the Secondary Site.

Since the connection to the Primary Site is lost, the operator automatically sends a GRPC request down to the driver to forcefully mark the dataSource as primary on the Secondary Site.

Prerequisites

  • The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

Procedures

  1. Create VolumeReplications for all the PVCs for which mirroring is enabled on the Secondary.

    Example
    cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  volumeReplicationClass: rbd-volumereplicationclass
  replicationState: primary
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <pvc-name>
EOF
    1. <vr-name>: The name of the VolumeReplication object, recommended to be the same as the PVC name.
    2. <namespace>: The namespace to which the VolumeReplication belongs, which must be the same as the PVC namespace.
    3. <pvc-name>: The name of the PVC for which Mirror needs to be enabled.

  2. Check VolumeReplication CR status to verify if the image is marked primary on the secondary site.

  3. Once the Image is marked as primary, the PVC is now ready to be used. Now, we can scale up the applications to use the PVC.

Failback (post-disaster recovery)

Once the failed cluster is recovered on the primary site and you want to failback from secondary site, follow the below steps:

Prerequisites

  • The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

Procedures

  1. Scale down the running applications (if any) on the primary site. Ensure that all persistent volumes in use by the workload are no longer in use on the primary cluster.

  2. Update VolumeReplication CR replicationState from primary to secondary on the primary site.

  3. Scale down the applications on the secondary site.

  4. Update VolumeReplication CR replicationState state from primary to secondary in secondary site.

  5. On the primary site, verify the VolumeReplication status is marked as volume ready to use.

  6. Once the volume is marked to ready to use, change the replicationState state from secondary to primary in primary site.

  7. Scale up the applications again on the primary site.