Alauda Container Platform

Backup and Restore of Prometheus Monitoring Data

TOC

Feature Overview

Prometheus monitoring data is stored in TSDB (Time Series Database) format, supporting backup and restore functionalities. The monitoring data is stored in a designated path within the Prometheus container (specified by the configuration storage.tsdb.path, which defaults to /prometheus).

template:
  spec:
    containers:
      - args:
          - '--storage.tsdb.path=/prometheus' # Directory for storing monitoring data in the Prometheus container

Use Cases

  • Retaining historical monitoring data during system migration
  • Preventing data loss due to unexpected incidents
  • Migrating monitoring data to a new Prometheus instance

Prerequisites

  • The ACP Monitoring with Prometheus plugin has been installed (the name of the compute component is prometheus-kube-prometheus-0, and the type is StatefulSet)
  • Administrator privileges for the cluster
  • Ensure there is sufficient storage space at the target storage location

Procedures to Operate

Backup Data

Before starting the backup, please note: When Prometheus stores monitoring data, it first places the collected data into a cache and then periodically writes it to the storage directory. The following backup methods use the storage directory as the data source, so they do not include the data in the cache at the time of backup.

  1. Use the kubectl cp command to back up:

    kubectl cp -n cpaas-system prometheus-kube-prometheus-0-0:/prometheus -c prometheus <target storage path>

  2. Backup from the storage backend (based on the type of storage selected during installation):

  • LocalVolume: Copy from the /cpaas/monitoring/prometheus directory
  • PV: Copy from the PV mount directory (it is recommended to set the PV's persistentVolumeReclaimPolicy to Retain)
  • StorageClass: Copy from the PV mount directory

Method 2: Snapshot Backup

  1. Enable Admin API:

    kubectl edit -n cpaas-system prometheus kube-prometheus-0

    Add the configuration:

    spec:
  enableAdminAPI: true

    Note: After updating and saving the configuration, the Prometheus Pod (Pod name: prometheus-kube-prometheus-0-0) will restart. Wait until all Pods are in Running status before proceeding with subsequent operations.

  2. Create a snapshot:

    curl -XPOST <Prometheus Pod IP>:9090/api/v1/admin/tsdb/snapshot

Restore Data

  1. Copy the backup data to the Prometheus container:

    kubectl cp ./prometheus-backup cpaas-system/prometheus-kube-prometheus-0-0:/prometheus/

  2. Move data into the specified directory:

    kubectl exec -it -n cpaas-system prometheus-kube-prometheus-0-0 -c prometheus sh
mv /prometheus/prometheus-backup/* /prometheus/

    Shortcut: When the storage type is LocalVolume during plugin installation, simply copy the backup data directly to the /cpaas/monitoring/prometheus/prometheus-db/ directory of the node where the plugin is installed.

Operation Results

  • After backup is complete, the complete TSDB format monitoring data can be seen at the target storage path
  • After restoration is complete, Prometheus will automatically load the historical monitoring data

Learn More

TSDB Data Format Description

Example of TSDB format data structure:

├── 01FXP317QBANGAX1XQAXCJK9DB
│   ├── chunks
│   │   └── 000001
│   ├── index
│   ├── meta.json
│   └── tombstones
├── chunks_head
│   ├── 000022
│   └── 000023
├── queries.active
└── wal
    ├── 00000020
    ├── 00000021
    ├── 00000022
    ├── 00000023
    └── checkpoint.00000019
        └── 00000000

Data Backup Considerations

  • Backup data does not include the cached data at the time of backup
  • It is recommended to perform data backups regularly
  • When using PV storage, it is advisable to set the persistentVolumeReclaimPolicy to Retain

Next Procedures

  • Verify whether the monitoring data has been correctly restored
  • Regularly schedule data backup plans
  • If using the snapshot backup method, you may opt to disable the Admin API