Alauda Container Platform

Control Access & Quotas for COSI Buckets with CephObjectStoreUser (Ceph Driver)

This guide shows Kubernetes administrators how to combine CephObjectStoreUser (COSU), BucketClass/BucketClaim, and BucketAccessClass/BucketAccess to implement least-privileged access and quota enforcement for Ceph RGW-backed COSI buckets.

What you'll build

  1. A CephObjectStoreUser with explicit capabilities and optional per-user quotas;
  2. A BucketClass that tells the Ceph COSI driver which COSU credentials to use;
  3. One or more BucketClaims to provision buckets;
  4. Fine-grained, per-workload credentials using BucketAccessClass/BucketAccess (read-only, write-only, read-write), with optional anonymous read.

TOC

Prerequisites

  • A healthy Ceph cluster with RGW and Rook installed.
  • COSI plugins installed.
  • Cluster admin privileges (to create cluster-scoped resources).

Step 1 — Create a CephObjectStoreUser (with capabilities & quotas)

CephObjectStoreUser is the service account the driver uses to perform bucket operations in RGW. It must live in the rook-ceph namespace and target your CephObjectStore.

apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: user-for-cosi
  namespace: rook-ceph
spec:
  # Target CephObjectStore
  store: object-store
  # Required capabilities for COSI bucket lifecycle
  capabilities:
    bucket: read, write
    user: read, write
  # Optional per-user quotas enforced by RGW
  quotas:
    maxBuckets: 50        # limit number of buckets this user can own
    maxObjects: 500        # total objects across buckets (example)
    maxSize: 100Gi       # total logical size across buckets
  displayName: "User for COSI driver"

Get the generated access keys (Rook creates a Secret for this user):

kubectl get cephobjectstoreuser user-for-cosi -n rook-ceph -o yaml
# Read status.info.secretName -> the Secret holding AccessKey/SecretKey

These COSU credentials are consumed by the driver (not by your apps) to create/delete buckets on your behalf.

Step 2 — Define a BucketClass bound to the CephObjectStoreUser

BucketClass instructs the driver which COSU Secret to use when provisioning buckets.

apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClass
metadata:
  name: ceph-cosi-driver-class
deletionPolicy: Delete
# Must match the driver name
driverName: ceph.objectstorage.k8s.io
parameters:
  # Reference the Secret created for the CephObjectStoreUser
  objectStoreUserSecretName: <secret-name-from-step-1>
  objectStoreUserSecretNamespace: rook-ceph

deletionPolicy controls the bucket's physical lifecycle when a BucketClaim is deleted (Delete vs Retain).

Step 3 — Provision a bucket with BucketClaim

Create the bucket in your application namespace by referencing the BucketClass.

apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClaim
metadata:
  name: my-bucket-claim
  namespace: app-a
spec:
  bucketClassName: ceph-cosi-driver-class
  # Required by the COSI API (choose the data API you need)
  protocols:
  - S3

Wait until status.bucketReady: true and note status.bucketName for the actual RGW bucket name.

Step 4 — Issue least-privileged credentials with BucketAccessClass/BucketAccess

Define policy templates with BucketAccessClass and mint credentials per workload via BucketAccess. Supported policies: readonly, writeonly, readwrite. Anonymous read is available by setting parameters.anonymous: "true" (string).

Example BucketAccessClass (read-only)

apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketAccessClass
metadata:
  name: ceph-readonly-access-class
authenticationType: KEY
driverName: ceph.objectstorage.k8s.io
parameters:
  policy: readonly
  anonymous: "false"

Mint credentials with BucketAccess

apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketAccess
metadata:
  name: my-bucket-readonly-access
  namespace: app-a
spec:
  bucketAccessClassName: ceph-readonly-access-class
  bucketClaimName: my-bucket-claim
  credentialsSecretName: my-bucket-readonly-credentials

The driver writes a Secret named in credentialsSecretName. Decode .data.BucketInfo (base64) to get secretS3.endpoint, accessKeyID, and accessSecretKey for your S3 client.

Tip: Issue distinct credentials per Deployment/Job to simplify rotation and revocation without disrupting other workloads.

Step 5 — Anonymous public read (optional)

If you need public static asset hosting:

apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketAccessClass
metadata:
  name: ceph-anonymous-readonly-class
authenticationType: KEY
driverName: ceph.objectstorage.k8s.io
parameters:
  policy: readonly
  anonymous: "true"

Bind it with a BucketAccess to your BucketClaim. Once granted, objects are retrievable via unauthenticated HTTP GET (ensure network exposure as appropriate).

Step 6 — Quota control: where to enforce and how to change

Scope: The quotas block on CephObjectStoreUser applies per user and is enforced by RGW across all buckets owned by that user.

  • maxBuckets: upper bound on buckets the user can create/own.
  • maxObjects: maximum number of objects the user can store (across buckets).
  • maxSize: total logical size permitted for the user.

Update quotas by editing the COSU resource:

kubectl -n rook-ceph edit cephobjectstoreuser user-for-cosi
# Modify spec.quotas.{maxBuckets,maxObjects,maxSize}; save and exit.
# Rook reconciles and applies the new RGW user quotas.

Design choice: Keep COSU quotas relatively tight to bound blast radius. Use least-privilege policies via BAC/BA to limit what a given set of application credentials can do within a bucket.

Operations & troubleshooting

  • Namespace placement: CephObjectStoreUser and its Secret must be in rook-ceph. App-level resources (BucketClaim, BucketAccess) should live in the app namespace.
  • Policy not applied: confirm bucketAccessClassName and parameters.policy in BAC (readonly|writeonly|readwrite).
  • Anonymous read fails: ensure anonymous: "true" is a string, not boolean; verify endpoint exposure and HTTP access path (/<bucket>/<object>).
  • Can't find keys: check the BucketAccess Secret and decode .data.BucketInfo.
  • Bucket stays not-ready: check controller/driver logs (e.g. kubectl -n cpaas-system logs deploy/ceph-cosi-driver -c ceph-cosi-driver).
  • Rotation: create a new BucketAccess, roll your workloads to the new Secret, then delete the old Secret/BA.

Cleanup

  • Remove a workload credential by deleting its BucketAccess and the referenced Secret.
  • Removing a bucket: delete the BucketClaim (behavior follows BucketClass.deletionPolicy). If the backend refuses deletion because the bucket is not empty, delete objects first.