Create Standard Type Cluster

A standard-type cluster is the most typical deployment method for Ceph storage. It distributes data replicas across hard drives on different hosts, ensuring that if a single host fails, the data copies on other hosts can still maintain service availability.

TOC

Prerequisites

Prepare Package

  • Download the Alauda Container Platform Storage Essentials installation package corresponding to your platform architecture.

  • Upload the Alauda Container Platform Storage Essentials installation package using the Upload Packages mechanism.

  • Download the Alauda Build of Rook-Ceph installation package corresponding to your platform architecture.

  • Upload the Alauda Build of Rook-Ceph installation package using the Upload Packages mechanism.

Prepare Infrastructure

  • At least 3 nodes are required in the storage cluster.

  • Each node must have at least 1 blank hard disk or 1 unformatted hard disk partition available.

  • The available hard disk capacity is recommended to be greater than 50 G.

  • If you are using an attached Kubernetes cluster with Containerd as the runtime component, please ensure that the LimitNOFILE parameter value in the /etc/systemd/system/containerd.service file is configured to 1048576 on all nodes of the cluster, to ensure successful deployment of distributed storage. For configuration instructions, please refer to Modifying Containerd Configuration Information.

    Note: When upgrading from versions earlier than v3.10.2 to the current version, if you need to deploy Ceph distributed storage on your custom Kubernetes cluster with Containerd as the runtime component, you must also set the LimitNOFILE parameter value in the /etc/systemd/system/containerd.service file to 1048576 on all nodes of the cluster.

Precautions

Creating Storage Service and Accessing Storage Service only support selecting one method.

Procedure

Deploy Alauda Container Platform Storage Essentials

  1. Login, go to the Administrator page.

  2. Click Marketplace > OperatorHub to enter the OperatorHub page.

  3. Find the Alauda Container Platform Storage Essentials, click Install, and navigate to the Install Alauda Container Platform Storage Essentials page.

    Configuration Parameters:

    ParameterRecommended Configuration
    ChannelThe default channel is stable.
    Installation ModeCluster: All namespaces in the cluster share a single Operator instance for creation and management, resulting in lower resource usage.
    Installation PlaceSelect Recommended, Namespace only support acp-storage.
    Upgrade StrategyManual: When there is a new version in the Operator Hub, manual confirmation is required to upgrade the Operator to the latest version.

(Optional) Deploy Alauda Build of LocalStorage

When utilizing the Selection Device method to add storage devices to your Ceph cluster, it is necessary to deploy the Alauda Build of LocalStorage Operator. This Operator is responsible for automatically discovering all hard disk devices across every node within the Kubernetes cluster and collecting comprehensive device information, thereby streamlining the storage integration process.

  1. Login, go to the Administrator page.

  2. Click Marketplace > OperatorHub to enter the OperatorHub page.

  3. Find the Alauda Build of LocalStorage, click Install, and navigate to the Install Alauda Build of LocalStorage page.

    Configuration Parameters:

    ParameterRecommended Configuration
    ChannelThe default channel is stable.
    Installation ModeCluster: All namespaces in the cluster share a single Operator instance for creation and management, resulting in lower resource usage.
    Installation PlaceSelect Recommended, Namespace only support acp-storage.
    Upgrade StrategyManual: When there is a new version in the Operator Hub, manual confirmation is required to upgrade the Operator to the latest version.

Deploy Operator

  1. Navigate to Administrator.

  2. In the left sidebar, click Storage Management > Distributed Storage.

  3. Click Configure Now.

  4. In the Deploy Operator wizard page, click the Deploy Operator button at the bottom right.

    • When the page automatically advances to the next step, it indicates that the Operator has been deployed successfully.

    • If the deployment fails, please refer to the prompt on the interface Clean Up Deployed Information and Retry, and redeploy the Operator; if you wish to return to the distributed storage selection page, click Application Store, first uninstall the resources in the already deployed rook-operator, and then uninstall rook-operator.

Create Cluster

  1. In the Create Cluster wizard page, configure the relevant parameters and click the Create Cluster button at the bottom right.

    ParameterExplanation
    Cluster TypeSelect Standard.
    Device Class TypeDevice classes are groupings of hard disks; you can customize device classes according to your storage needs, allocating different storage content to disks of varying performance.
    • Default Device Class: The platform will automatically categorize the types of hard disks in the cluster nodes. For instance, creating device classes named hdd, ssd, nvme.
    • Custom Device Class: Customize the name of the device class for specific combinations of disks in the node; adding multiple device classes is supported. The same hard disk can only belong to one device class.
    Device Class - NameThe name of the device class. When selecting Custom Device Class, the device class cannot use the following names: hdd, ssd, nvme.
    Device Class - Storage DevicesTo add storage devices to a device class, you can choose between the Selection Device and Input Device methods:
    • Selection Device:
      Select from available storage devices. A device is considered available if it meets the following criteria:
      • Device type is either disk or mpath
      • No file system is detected (fsType is blank)
      • Capacity exceeds 10 GiB
      Devices such as rbd, nbd, and dm-* will not be displayed in the list of available selectable devices.

      Note: Requires prior deployment of the Alauda Build of LocalStorage Operator.
    • Input Device:
      Manually input the names of the blank devices under the node, such as sda.

      Note: For optimal performance and management, it is strongly advised to utilize raw disks as storage devices instead of employing individual partitions on a disk.
    SnapshotWhen enabled, it supports creating PVC snapshots and using snapshots to configure new PVCs for quick backup and recovery of business data.
    If you did not enable snapshots when creating storage, you can still enable them as needed from the Operations section on the storage cluster details page.
    Note: Please ensure that you have deployed volume snapshot plugins for the current cluster before using.
    Monitoring AlarmWhen enabled, it will provide out-of-the-box monitoring metric collection and alerting capabilities, see Monitoring and Alarming.
    Note: If not enabled at this time, you will need to find alternative solutions for storage monitoring and alarms. For example, manually configuring monitoring dashboards and alert strategies in the operation and maintenance center.
  2. Click Advanced Configuration for advanced component configuration.

    ParameterExplanation
    Network Configuration
    • Host Network: The storage cluster will use the host network, and you should fill in the relevant network optimization parameters in the optimization parameters column, such as configuring the public and cluster subnets. If left blank, the default host subnet will be used.
      Note: Using the host network may expose security risks due to unencrypted (plaintext) transmission of data through host ports. Please contact the platform support team to obtain the encrypted transmission solution.
    • Container Network: The storage cluster will use container networking; you can create subnets in network management and assign them to the rook-ceph namespace. If left blank, the default subnet will be used.
      Note:
      IPv6 not supported.
      When using the container network, storage is only accessible within the cluster.
      Failures or restarts of the Ceph CSI Pod may result in service interruptions.
    Optimization ParametersSupports filling parameters in Ceph configuration file format; the system will overwrite the default parameters based on the provided content.
    Note: After first filling in or modifying initialization parameters, please click on the initialization parameters; successful initialization is required before a cluster can be created.
    Component Fixed-point DeploymentYou can deploy components to specified nodes; at least three nodes are required to ensure minimum availability. The components eligible for fixed-point deployment configuration include MON, MGR, MDS, RGW.
    • When the page automatically advances to the next step, it indicates that the Ceph cluster has been deployed successfully.

    • If the creation fails, you may click to clean up Created Information or Retry to automatically clean up the resources and recreate the cluster, or manually clean up resources according to the documentation Distributed Storage Service Resource Cleanup.

Create Storage Pool

  1. In the Create Storage Pool wizard page, configure the relevant parameters and click the Create Storage Pool button at the bottom right.

    ParameterExplanation
    Storage Type
    • File Storage: Provides secure, reliable, and scalable shared file storage services. Suitable for file sharing, data backup, etc.
    • Block Storage: Provides high IOPS and low-latency storage services. Suitable for databases, virtualization, etc.
    • Object Storage: Provides standard S3 interface storage services, suitable for big data, backup archiving, cloud storage, etc.
    Replica CountThe larger the number of replicas, the higher the redundancy and data security; however, the utilization rate of storage will decrease. It is usually set to 3 to meet most needs.
    Device ClassUniformly classify types for the same type of device or disks of the same business logic, selecting from the device classes added in the previous step.
    • When selecting a device class, data will be stored in the chosen device class.
    • If no device class is selected, data will be randomly stored across all devices in the storage pool.

    If it is object storage, you also need to configure the following parameters:

    ParameterExplanation
    RegionSpecify the region where the storage pool is located.
    Gateway TypeDefault is S3 and cannot be modified.
    Internal PortSpecify the port for internal access in the cluster.
    External AccessEnabling/disabling external access will create/destroy Nodeport type Service.
    Instance CountThe number of resource instances for object storage.
    • When the page automatically advances to the next step, it indicates that the storage pool has been deployed successfully.

    • If the deployment fails, please refer to the interface prompts to check the core components, and then click Clean Up Created Information and Retry to recreate the storage pool.

  2. Click Create Storage Pool. In the Details tab, you can view information about the created storage pool.

Create Stretch Type Cluster

For details, please refer to Create Stretch Type Cluster.

Cleanup Distributed Storage

For details, please refer to Cleanup Distributed Storage.