Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Kopia] Backup partially failing with timeout error after 24 hours despite default timeout of 4h #8220

Open
navilg opened this issue Sep 16, 2024 · 10 comments
Open

[Kopia] Backup partially failing with timeout error after 24 hours despite default timeout of 4h #8220

navilg opened this issue Sep 16, 2024 · 10 comments
Assignees
@blackpiglet
Labels
Kopia Needs investigation PodVolume

Comments

@navilg
Copy link

navilg commented Sep 16, 2024 •
edited by blackpiglet
Loading

What steps did you take and what happened:

A scheduled backup is running.
Backup get stuck for 24 hours and fails with timeout error.
In backup description I see some pod volumes backup completed but many missing. They are not even listed in failed or new section of description.

Logs:

<details>
  <summary>
    Expand for logs
  </summary>
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otac-0
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="timed out waiting for all PodVolumeBackups to complete" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/podvolume/backupper.go:317" error.function="github.com/vmware-tanzu/velero/pkg/podvolume.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:448" name=otac-0
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-2wbc8
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-2wbc8
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-2xjrt
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-2xjrt
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-4cmnl
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-4cmnl
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-7bt6z
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-7bt6z
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-7cks5
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-7cks5
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-7h77k
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-7h77k
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-9bxjs
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-9bxjs
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-9hs7f
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-9hs7f
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-d8244
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-d8244
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-dfhhm
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-dfhhm
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-f4wzm
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-f4wzm
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-fjp2m
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-fjp2m
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-fx6n2
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-fx6n2
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-gmr2p
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-gmr2p
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-hfpcr
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-hfpcr
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-hms76
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-hms76
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-hpxfl
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-hpxfl
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-kvhdm
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-kvhdm
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-l82dv
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-l82dv
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-mhlsb
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-mhlsb
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-mt42z
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-mt42z
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-nzp95
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-nzp95
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-qn47w
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-qn47w
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-qn9jw
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-qn9jw
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-stw2s
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-stw2s
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-svhx2
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-svhx2
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-t4m9m
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-t4m9m
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-admin-0
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-admin-0
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-0
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-0
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-1
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-1
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-10
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-10
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-11
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-11
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-12
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-12
time="2024-09-15T13:29:07Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-13
time="2024-09-15T13:29:07Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-13
time="2024-09-15T13:29:07Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-14
time="2024-09-15T13:29:07Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-14
time="2024-09-15T13:29:07Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-15
time="2024-09-15T13:29:07Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-15
time="2024-09-15T13:29:07Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-16
time="2024-09-15T13:29:07Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-16
time="2024-09-15T13:29:08Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-17
time="2024-09-15T13:29:08Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-17
time="2024-09-15T13:29:08Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-18
time="2024-09-15T13:29:08Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-18
time="2024-09-15T13:29:08Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-19
time="2024-09-15T13:29:08Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-19
time="2024-09-15T13:29:09Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-2
time="2024-09-15T13:29:09Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-2
time="2024-09-15T13:29:09Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-20
time="2024-09-15T13:29:09Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-20
time="2024-09-15T13:29:09Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-21
time="2024-09-15T13:29:09Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-21
time="2024-09-15T13:29:10Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-22
time="2024-09-15T13:29:10Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-22
time="2024-09-15T13:29:10Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-23
time="2024-09-15T13:29:10Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-23
time="2024-09-15T13:29:10Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-24
time="2024-09-15T13:29:10Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-24
time="2024-09-15T13:29:10Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-25
time="2024-09-15T13:29:10Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-25
time="2024-09-15T13:29:11Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-26
time="2024-09-15T13:29:11Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-26
time="2024-09-15T13:29:11Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-27
time="2024-09-15T13:29:11Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-27
time="2024-09-15T13:29:11Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-3
time="2024-09-15T13:29:11Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-3
time="2024-09-15T13:29:12Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-4
time="2024-09-15T13:29:12Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-4
time="2024-09-15T13:29:12Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-5
time="2024-09-15T13:29:12Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-5
time="2024-09-15T13:29:12Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-6
time="2024-09-15T13:29:12Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-6
time="2024-09-15T13:29:13Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-7
time="2024-09-15T13:29:13Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-7
time="2024-09-15T13:29:13Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-8
time="2024-09-15T13:29:13Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-8
time="2024-09-15T13:29:13Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-9
time="2024-09-15T13:29:13Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-9
time="2024-09-15T13:29:13Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-0
time="2024-09-15T13:29:13Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-0
time="2024-09-15T13:29:14Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-1
time="2024-09-15T13:29:14Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-1
time="2024-09-15T13:29:14Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-10
time="2024-09-15T13:29:14Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-10
time="2024-09-15T13:29:14Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-11
time="2024-09-15T13:29:14Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-11
time="2024-09-15T13:29:15Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-12
time="2024-09-15T13:29:15Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-12
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-13
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-13
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-14
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-14
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-15
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-15
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-16
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-16
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-17
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-17
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-18
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-18
time="2024-09-15T13:29:17Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-19
time="2024-09-15T13:29:17Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-19
time="2024-09-15T13:29:17Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-2
time="2024-09-15T13:29:17Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-2
time="2024-09-15T13:29:17Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-20
time="2024-09-15T13:29:17Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-20
time="2024-09-15T13:29:18Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-21
time="2024-09-15T13:29:18Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-21
time="2024-09-15T13:29:18Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-22
time="2024-09-15T13:29:18Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-22
time="2024-09-15T13:29:18Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-23
time="2024-09-15T13:29:18Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-23
time="2024-09-15T13:29:19Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-24
time="2024-09-15T13:29:19Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-24
time="2024-09-15T13:29:19Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-25
time="2024-09-15T13:29:19Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-25
time="2024-09-15T13:29:19Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-26
time="2024-09-15T13:29:19Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-26
time="2024-09-15T13:29:19Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-3
time="2024-09-15T13:29:19Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-3
time="2024-09-15T13:29:20Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-4
time="2024-09-15T13:29:20Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-4
time="2024-09-15T13:29:20Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-5
time="2024-09-15T13:29:20Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-5
time="2024-09-15T13:29:20Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-6
time="2024-09-15T13:29:20Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-6
time="2024-09-15T13:29:21Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-7
time="2024-09-15T13:29:21Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-7
time="2024-09-15T13:29:21Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-8
time="2024-09-15T13:29:21Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-8
time="2024-09-15T13:29:21Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-9
time="2024-09-15T13:29:21Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-9
time="2024-09-15T13:29:21Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otiv-amqp-0
time="2024-09-15T13:29:21Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otiv-amqp-0
time="2024-09-15T13:29:22Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otiv-amqp-1
time="2024-09-15T13:29:22Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otiv-amqp-1
time="2024-09-15T13:29:22Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otiv-amqp-2
time="2024-09-15T13:29:22Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otiv-amqp-2
time="2024-09-15T13:29:22Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otiv-amqp-3
time="2024-09-15T13:29:22Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otiv-amqp-3
time="2024-09-15T13:29:24Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=trident-controller-6b96f5dbcf-m4lqb
time="2024-09-15T13:29:24Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=trident-controller-6b96f5dbcf-m4lqb
</details>

What did you expect to happen:

Backup to work fine.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

  • kubectl logs deployment/velero -n velero
  • velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
  • velero backup logs <backupname>
  • velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
  • velero restore logs <restorename>

Anything else you would like to add:

Environment:

  • Velero version (use velero version): 1.12.3
  • Velero features (use velero client config get features): None
  • Kubernetes version (use kubectl version): 1.28
  • Kubernetes installer & version: GKE 1.28
  • Cloud provider or hardware configuration: GCP
  • OS (e.g. from /etc/os-release): Ubuntu 22.04 with containerd

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

  • 👍 for "I would like to see this bug fixed as soon as possible"
  • 👎 for "There are more important bugs to focus on right now"
@navilg
Copy link
Author

navilg commented Sep 16, 2024

After velero pod restart it start working fine. But after few days, It again start failing.

@Lyndon-Li
Copy link
Contributor

Please share Velero log bundle by running velero debug

@navilg
Copy link
Author

navilg commented Sep 18, 2024

Thanks @Lyndon-Li
Attached.

bundle-2024-09-18-09-53-00.tar.gz

@blackpiglet
Copy link
Contributor

Could you try with v1.14.1?
It seems related to PodVolumeBackup's event handler is not removed after the PodVolumeBackup completion.
That is fixed in release-1.14.

@blackpiglet blackpiglet self-assigned this Sep 23, 2024
@navilg
Copy link
Author

navilg commented Sep 23, 2024 •
edited
Loading

Thanks @blackpiglet Upgrading velero to 1.14.0 is in my plan. is the same fix available in this as well ?

@blackpiglet
Copy link
Contributor

It should be, but v1.14.1 contains some other fixes. It's better to use the latest patch release.

@blackpiglet
Copy link
Contributor

blackpiglet commented Sep 23, 2024 •
edited
Loading

After investigation, the timeout after 24 hours issue was not caused by the PodVolumeBackup's event handler's de-registered issue.

time="2024-09-16T14:00:29Z" level=info msg="pod eim1/otawg-deployment-bf74b684f-2wbc8 has volumes to backup: [apps-volume]" backup=velero/sch-backup-eim1-prod-daily-20240916135436 logSource="pkg/podvolume/backupper.go:174" name=otawg-deployment-bf74b684f-2wbc8 namespace=eim1 resource=pods
time="2024-09-17T14:04:59Z" level=warning msg="volume apps-volume is declared in pod eim1/otawg-deployment-bf74b684f-2wbc8 but not mounted by any container, skipping" backup=velero/sch-backup-eim1-prod-daily-20240916135436 logSource="pkg/podvolume/backupper.go:284" name=otawg-deployment-bf74b684f-2wbc8 namespace=eim1 resource=pods

The 24-hour gap happens here.
The Velero server pod took 24 hours to determine the volume apps-volume should not be handled by PodVolumeBackup.

I haven't found any logic that would consume that much time yet.

@navilg
Could you help to check what happened between 2024-09-16T14:00:29Z and 2024-09-17T14:04:59Z in the k8s environment?

@navilg
Copy link
Author

navilg commented Sep 23, 2024

Let me take a check about that volume

@navilg
Copy link
Author

navilg commented Sep 24, 2024

app-volume is an emptyDir volume. I will test after excluding this volume from backup. Do we know of any known issue with backing-up of emptyDir volumes ?
Also, is there a way to exclude all emptyDir volumes from backup other than annotating each pods using that volume ?

@blackpiglet
Copy link
Contributor

PodVolumeBackup can work with the emptyDir type of volume.
https://velero.io/docs/v1.12/resource-filtering/#resource-policies
Yes, there is a way to exclude volumes from backup by type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@blackpiglet blackpiglet

Labels
Kopia Needs investigation PodVolume
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@navilg @blackpiglet @Lyndon-Li