Skip to content

K8SPSMDB-1072: add retention section to backup tasks #1928

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Jun 2, 2025
Merged

K8SPSMDB-1072: add retention section to backup tasks #1928

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8729890
K8SPSMDB-1072: add `retention` section to backup tasks
pooknull May 16, 2025
f9b086f
add retention to tests
pooknull May 16, 2025
6ab6283
fix unit test
pooknull May 19, 2025
1436357
remove unused comment
pooknull May 19, 2025
29dfe95
Merge branch 'main' into K8SPSMDB-1072
pooknull May 19, 2025
b7f0ce5
Merge branch 'main' into K8SPSMDB-1072
hors May 21, 2025
deb4d75
Merge branch 'main' into K8SPSMDB-1072
pooknull May 22, 2025
c87ce9f
fix test
pooknull May 23, 2025
e9b7656
Merge branch 'main' into K8SPSMDB-1072
pooknull May 26, 2025
1baebc8
small improvement
pooknull May 27, 2025
21f5606
fix copypaste
pooknull May 27, 2025
4b5f654
fix lint
pooknull May 27, 2025
be2e997
Merge remote-tracking branch 'origin/main' into K8SPSMDB-1072
pooknull May 27, 2025
d21f06a
fix bundle
pooknull May 27, 2025
2f867bd
fix tests
pooknull May 28, 2025
c8e7a48
Merge remote-tracking branch 'origin/main' into K8SPSMDB-1072
pooknull May 28, 2025
01ad2b3
Merge branch 'main' into K8SPSMDB-1072
hors May 28, 2025
4ba045e
fix test
pooknull May 29, 2025
4ea859d
Merge branch 'main' into K8SPSMDB-1072
hors Jun 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions config/crd/bases/psmdb.percona.com_perconaservermongodbs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,17 @@ spec:
type: integer
name:
type: string
retention:
properties:
count:
type: integer
deleteFromStorage:
type: boolean
type:
enum:
- count
type: string
type: object
schedule:
type: string
storageName:
Expand Down
11 changes: 11 additions & 0 deletions deploy/bundle.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1108,6 +1108,17 @@ spec:
type: integer
name:
type: string
retention:
properties:
count:
type: integer
deleteFromStorage:
type: boolean
type:
enum:
- count
type: string
type: object
schedule:
type: string
storageName:
Expand Down
20 changes: 16 additions & 4 deletions deploy/cr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,21 +709,30 @@ spec:
# - name: daily-s3-us-west
# enabled: true
# schedule: "0 0 * * *"
# keep: 3
# retention:
# count: 3
# type: count
# deleteFromStorage: true
# storageName: s3-us-west
# compressionType: gzip
# compressionLevel: 6
# - name: weekly-s3-us-west
# enabled: false
# schedule: "0 0 * * 0"
# keep: 5
# retention:
# count: 3
# type: count
# deleteFromStorage: true
# storageName: s3-us-west
# compressionType: gzip
# compressionLevel: 6
# - name: weekly-s3-us-west-physical
# enabled: false
# schedule: "0 5 * * 0"
# keep: 5
# retention:
# count: 3
# type: count
# deleteFromStorage: true
# type: physical
# storageName: s3-us-west
# compressionType: gzip
Expand All @@ -738,7 +747,10 @@ spec:
# - name: weekly-s3-us-west-incremental-base
# enabled: false
# schedule: "0 5 * * 0"
# keep: 5
# retention:
# count: 3
# type: count
# deleteFromStorage: true
# type: incremental-base
# storageName: s3-us-west
# compressionType: gzip
Expand Down
11 changes: 11 additions & 0 deletions deploy/crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1108,6 +1108,17 @@ spec:
type: integer
name:
type: string
retention:
properties:
count:
type: integer
deleteFromStorage:
type: boolean
type:
enum:
- count
type: string
type: object
schedule:
type: string
storageName:
Expand Down
11 changes: 11 additions & 0 deletions deploy/cw-bundle.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1108,6 +1108,17 @@ spec:
type: integer
name:
type: string
retention:
properties:
count:
type: integer
deleteFromStorage:
type: boolean
type:
enum:
- count
type: string
type: object
schedule:
type: string
storageName:
Expand Down
15 changes: 10 additions & 5 deletions e2e-tests/scheduled-backup/conf/some-name-rs0-2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,31 +49,36 @@ spec:
tasks:
- name: every-min-minio
enabled: true
keep: 1
retention:
count: 1
schedule: "* * * * *"
compressionType: gzip
storageName: minio
- name: every-min-aws-s3
enabled: true
keep: 1
retention:
count: 1
schedule: "* * * * *"
compressionType: gzip
storageName: aws-s3
- name: every-min-gcp-cs
enabled: true
keep: 1
retention:
count: 1
schedule: "* * * * *"
compressionType: gzip
storageName: gcp-cs
- name: every-min-azure-blob
enabled: true
keep: 1
retention:
count: 1
schedule: "* * * * *"
compressionType: gzip
storageName: azure-blob
- name: weekly
enabled: true
keep: 1
retention:
count: 1
schedule: "0 0 * * 0"
compressionType: gzip
storageName: aws-s3
Expand Down
5 changes: 4 additions & 1 deletion e2e-tests/scheduled-backup/conf/some-name-rs0-3.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,16 @@ spec:
tasks:
- name: every-min-minio
enabled: true
keep: 1
retention:
count: 1
type: physical
schedule: "* * * * *"
compressionType: gzip
storageName: minio
- name: weekly
enabled: true
retention:
count: 1
schedule: "0 0 * * 0"
compressionType: gzip
storageName: aws-s3
Expand Down
11 changes: 11 additions & 0 deletions e2e-tests/version-service/conf/crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1108,6 +1108,17 @@ spec:
type: integer
name:
type: string
retention:
properties:
count:
type: integer
deleteFromStorage:
type: boolean
type:
enum:
- count
type: string
type: object
schedule:
type: string
storageName:
Expand Down
23 changes: 22 additions & 1 deletion pkg/apis/psmdb/v1/psmdb_defaults.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/utils/ptr"
logf "sigs.k8s.io/controller-runtime/pkg/log"

"github.com/percona/percona-backup-mongodb/pbm/compress"
Expand Down Expand Up @@ -514,10 +515,30 @@ func (cr *PerconaServerMongoDB) CheckNSetDefaults(ctx context.Context, platform
}

if cr.Spec.Backup.Enabled {
for _, bkpTask := range cr.Spec.Backup.Tasks {
for i := range cr.Spec.Backup.Tasks {
bkpTask := &cr.Spec.Backup.Tasks[i]

if bkpTask.Name == "" {
return errors.Errorf("backup task %d should have a name", i)
}
if string(bkpTask.CompressionType) == "" {
bkpTask.CompressionType = compress.CompressionTypeGZIP
}

if cr.CompareVersion("1.21.0") >= 0 {
if bkpTask.Keep > 0 && bkpTask.Retention.Count == 0 {
log.Info(".spec.backup.tasks[].keep will be deprecated in the future. Consider using .spec.backup.tasks[].retention.count instead", "task", bkpTask.Name)
continue
}

if bkpTask.Retention.Type == "" {
bkpTask.Retention.Type = BackupTaskSpecRetentionTypeCount
}

if bkpTask.Retention.DeleteFromStorage == nil {
bkpTask.Retention.DeleteFromStorage = ptr.To(true)
}
}
}
if len(cr.Spec.Backup.ServiceAccountName) == 0 && cr.CompareVersion("1.15.0") < 0 {
cr.Spec.Backup.ServiceAccountName = "percona-server-mongodb-operator"
Expand Down
14 changes: 14 additions & 0 deletions pkg/apis/psmdb/v1/psmdb_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -921,6 +921,7 @@ type BackupTaskSpec struct {
Name string `json:"name"`
Enabled bool `json:"enabled"`
Keep int `json:"keep,omitempty"`
Copy link
Contributor

@gkech gkech May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mark keep as deprecated?

Copy link
Contributor Author

@pooknull pooknull May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1baebc8

Retention BackupTaskSpecRetention `json:"retention,omitempty"`
Schedule string `json:"schedule,omitempty"`
StorageName string `json:"storageName,omitempty"`
CompressionType compress.CompressionType `json:"compressionType,omitempty"`
Expand All @@ -930,6 +931,19 @@ type BackupTaskSpec struct {
Type defs.BackupType `json:"type,omitempty"`
}

type BackupTaskSpecRetentionType string

const (
BackupTaskSpecRetentionTypeCount = "count"
)

type BackupTaskSpecRetention struct {
Count int `json:"count,omitempty"`
// +kubebuilder:validation:Enum={count}
Type string `json:"type,omitempty"`
DeleteFromStorage *bool `json:"deleteFromStorage,omitempty"`
}

Copy link
Contributor

@gkech gkech May 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that we could have a 1-1 mapping from keep to retention without using keep at all in our codebase in a similar way we implemented it in PXC. This way, when we will officially deprecate the keep option, we will not have to change the actual logic.

Check here: percona/percona-xtradb-cluster-operator@1553be6#diff-012d1e07c013a6332356a3ab856e8b0db6825aa9021d56c68ea5114faee14705

Copy link
Contributor Author

@pooknull pooknull May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1baebc8

func (task *BackupTaskSpec) JobName(cr *PerconaServerMongoDB) string {
return fmt.Sprintf("%s-backup-%s-%s", cr.Name, task.Name, cr.Namespace)
}
Expand Down
21 changes: 21 additions & 0 deletions pkg/apis/psmdb/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading