Skip to content

Issue with S3 Backup in Multi‑Repository pgBackRest Configuration #4469

Description

@Eric-zch

Overview
An ERROR: [032]: key '2' is not valid for 'repo' option occurred when using S3 for backup storage.

Environment
Please provide the following details:

Platform: Openshift
Platform Version: v4.16.37
PGO Image Tag: ubi8-5.7.9-0
Postgres Version: ubi8-14.15

Steps to Reproduce.
Yaml file for the postgrescluster

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: lpcs
spec:
  postgresVersion: 14
  openshift: true
  instances:
    - name: instance1
      replicas: 2
      minAvailable: 1
      resources:
        limits:
          cpu: 1000m
          memory: 1Gi
        requests:
          cpu: 1000m
          memory: 1Gi
      dataVolumeClaimSpec:
        accessModes:
        - "ReadWriteOnce"
        resources:
          requests:
            storage: 10Gi
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                topologyKey: topology.kubernetes.io/zone
                labelSelector:
                  matchLabels:
                    postgres-operator.crunchydata.com/cluster: lpcs
                    postgres-operator.crunchydata.com/instance-set: instance1
  backups:
    pgbackrest:
      configuration:
      - secret:
          name: ibm-cos-creds
      global:
        log-level-console: info
        repo1-retention-full: "10"
        repo1-retention-full-type: time
        repo2-path: "/ibas/dev-lpcs/lpcs/repo2"
        repo2-retention-full: "10"
        repo2-retention-full-type: time
      repos:
      - name: repo1
        schedules:
          full: "30 3 * * *"
        volume:
          volumeClaimSpec:
            accessModes:
            - "ReadWriteOnce"
            resources:
              requests:
                storage: 10Gi
      - name: repo2
        schedules:
          full: 0 3 * * *
        s3:
          bucket: bucket-db
          endpoint: https://s3.eu-de.cloud-object-storage.appdomain.cloud
          region: eu-de

Failed to backup to S3

➜  oc pgo backup lpcs --repoName=repo2 --options="--type=full"
postgresclusters/lpcs backup initiated
➜  oc get po
NAME                              READY   STATUS      RESTARTS   AGE
lpcs-backup-5257-2tp6l            0/1     Completed   0          2m46s
lpcs-backup-7j7x-bmrq7            0/1     Error       0          21s
lpcs-backup-7j7x-t7l4j            0/1     Error       0          48s
lpcs-instance1-5vv4-0             5/5     Running     0          3m14s
lpcs-instance1-pzxn-0             5/5     Running     0          3m14s
lpcs-pgbouncer-698d4b9979-fhzpc   2/2     Running     0          3m14s
lpcs-pgbouncer-698d4b9979-wnlcm   2/2     Running     0          3m14s
lpcs-repo-host-0                  2/2     Running     0          3m14s
➜  oc logs lpcs-backup-7j7x-t7l4j
2026-04-11 16:59:00.715 P00   INFO: backup command begin 2.54.1: --exec-id=1-7064a332 --log-level-console=info --log-level-file=off --pg1-host=lpcs-instance1-5vv4-0.lpcs-pods.dev-lpcs.svc.cluster.local. --pg2-host=lpcs-instance1-pzxn-0.lpcs-pods.dev-lpcs.svc.cluster.local. --pg1-host-ca-file=/etc/pgbackrest/conf.d/~postgres-operator/tls-ca.crt --pg2-host-ca-file=/etc/pgbackrest/conf.d/~postgres-operator/tls-ca.crt --pg1-host-cert-file=/etc/pgbackrest/conf.d/~postgres-operator/client-tls.crt --pg2-host-cert-file=/etc/pgbackrest/conf.d/~postgres-operator/client-tls.crt --pg1-host-key-file=/etc/pgbackrest/conf.d/~postgres-operator/client-tls.key --pg2-host-key-file=/etc/pgbackrest/conf.d/~postgres-operator/client-tls.key --pg1-host-type=tls --pg2-host-type=tls --pg1-path=/pgdata/pg14 --pg2-path=/pgdata/pg14 --pg1-port=5432 --pg2-port=5432 --pg1-socket-path=/tmp/postgres --pg2-socket-path=/tmp/postgres --repo=2 --repo2-path=/ibas/dev-lpcs/lpcs/repo2 --repo1-retention-full=10 --repo2-retention-full=10 --repo1-retention-full-type=time --repo2-retention-full-type=time --repo2-s3-bucket=bucket-db --repo2-s3-endpoint=https://s3.eu-de.cloud-object-storage.appdomain.cloud --repo2-s3-key=<redacted> --repo2-s3-key-secret=<redacted> --repo2-s3-region=eu-de --repo2-type=s3 --stanza=db --type=full
2026-04-11 16:59:00.835 P00   INFO: execute non-exclusive backup start: backup begins after the next regular checkpoint completes
2026-04-11 16:59:15.281 P00   INFO: backup start archive = 000000010000000000000007, lsn = 0/7000028
2026-04-11 16:59:15.281 P00   INFO: check archive for prior segment 000000010000000000000006
2026-04-11 16:59:16.102 P00  ERROR: [032]: local-1 process terminated unexpectedly [32]: ERROR: [032]: key '2' is not valid for 'repo' option
2026-04-11 16:59:16.102 P00   INFO: backup command end: aborted with exception [032]

Activity

  1. Eric-zch commented on Apr 11, 2026

    @Eric-zch
    Author

    The backup executes without issues when configured with a single repository.
    Interestingly, when I configure S3 as repo1 and local PVC as repo2, the backup completes successfully.
    The backup operates without errors using the YAML configuration shown below.

    apiVersion: postgres-operator.crunchydata.com/v1beta1
    kind: PostgresCluster
    metadata:
      name: lpcs
    spec:
      postgresVersion: 14
      openshift: true
      instances:
        - name: instance1
          replicas: 2
          minAvailable: 1
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi
            requests:
              cpu: 1000m
              memory: 1Gi
          dataVolumeClaimSpec:
            accessModes:
            - "ReadWriteOnce"
            resources:
              requests:
                storage: 10Gi
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 1
                  podAffinityTerm:
                    topologyKey: topology.kubernetes.io/zone
                    labelSelector:
                      matchLabels:
                        postgres-operator.crunchydata.com/cluster: lpcs
                        postgres-operator.crunchydata.com/instance-set: instance1
      backups:
        pgbackrest:
          configuration:
          - secret:
              name: ibm-cos-creds
          global:
            repo1-path: "/ibas/dev-lpcs/lpcs/repo1"
            repo1-retention-full: "10"
            repo1-retention-full-type: time
            repo2-retention-full: "10"
            repo2-retention-full-type: time
          image: de.icr.io/crunchy_pg_bhs_ns/crunchy-pgbackrest:ubi8-2.54.1-0
          repos:
          - name: repo1
            schedules:
              full: 0 3 * * *
            s3:
              bucket: bucket-db
              endpoint: https://s3.eu-de.cloud-object-storage.appdomain.cloud
              region: eu-de
          - name: repo2
            schedules:
              full: "30 3 * * *"
            volume:
              volumeClaimSpec:
                accessModes:
                - "ReadWriteOnce"
                resources:
                  requests:
                    storage: 10Gi
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions