feat: delete jobs in batches

[gaius-qi]

Description

This pull request introduces the concept of batch processing for garbage collection (GC) jobs in the manager/config package. The most important changes include adding a batch size parameter to the GC configuration, updating the validation logic to require this parameter, and implementing the batch deletion logic in the GC job.

Configuration updates:

• manager/config/config.go: Added BatchSize parameter to GCConfig struct and set its default value in the New function. Updated the Validate function to check for the presence of the BatchSize parameter. [1] [2] [3]

• manager/config/constants.go: Defined the DefaultJobGCBatchSize constant.

Testing updates:

• manager/config/config_test.go: Updated tests to include the BatchSize parameter and added a new test case to validate its presence. [1] [2]

Implementation updates:

• manager/job/gc.go: Replaced single deletion logic with deleteInBatches function to handle batch deletions. [1] [2]

Configuration file updates:

• manager/config/testdata/manager.yaml: Added batchSize parameter to the GC job configuration.

Related Issue

Motivation and Context

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation Update (if none of the other choices apply)

Checklist

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[codecov]

Codecov Report

Attention: Patch coverage is 35.29412% with 11 lines in your changes missing coverage. Please review.

Project coverage is 35.36%. Comparing base (6695100) to head (92bc021).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
manager/job/gc.go	0.00%	11 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3682   +/-   ##
=======================================
  Coverage   35.36%   35.36%           
=======================================
  Files         335      335           
  Lines       37617    37631   +14     
=======================================
+ Hits        13305    13310    +5     
- Misses      23483    23493   +10     
+ Partials      829      828    -1     

Flag	Coverage Δ
unittests	35.36% <35.29%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
manager/config/config.go	96.40% <100.00%> (+0.05%)	⬆️
manager/job/gc.go	0.00% <0.00%> (ø)

... and 3 files with indirect coverage changes

[chlins]

lgtm