ci: prune old artifacts on cluster lustre during weekly/release runs

[ko3n1g]

Claude summary

Summary

• Adds functional:cleanup_dgx_{a100,h100,gb200} jobs (one per cluster) to .gitlab/stages/04.functional-tests.yml, gated on FUNCTIONAL_TEST_SCOPE in [release, weekly]. Each submits a JET workload that deletes weekly-* and release-* directories older than 28 days at depth 1 of each cluster-side base dir.
• New JET cluster configs in dl/jet/ci on branches mcore/{coreweave,oci-hsg,draco-oci-iad,eos}-cpu — partition cpu (Eos: interactive, no dedicated CPU partition), no gpus_per_node, no exclusive. The cleanup never reserves a GPU.

How it works

Each functional:run_* job has an optional: true needs: on the matching functional:cleanup_dgx_<platform>:

• mr / nightly pipelines — cleanup excluded by rule; functional run starts immediately.
• weekly / release pipelines — cleanup runs first, then functional jobs.

.functional_cleanup derives CLEANUP_CLUSTER="cpu_${GPU_CLUSTER#dgx*_}" and submits via launch_jet_workload.py --scope cleanup. The launch script forwards CLEANUP_PATHS into the JET workload environment.

The recipe walks each CLEANUP_PATHS base dir at depth 1 only and matches directories whose names start with weekly- or release-. Matched directories older than 28 days are removed with rm -rf. The name pattern is the safety boundary — nothing outside those two prefixes is touched.

Required GitLab CI variables (already configured)

Set in project Settings → CI/CD → Variables (newline-separated paths, raw: true):

Variable	Purpose
CLEANUP_PATHS_DGX_{A100,H100,GB200}	Base dirs whose direct weekly-* / release-* children are eligible

Recipe exits cleanly with "nothing to do" if CLEANUP_PATHS is empty.

Recipe example

# tests/test_utils/recipes/_cleanup.yaml
spec:
  model: cleanup
  build: mcore-pyt-{environment}
  gpus: 0                       # CPU partition handles the rest
  script: |-
    THRESHOLD_DAYS=28
    while IFS= read -r path; do
        find "$path" -mindepth 1 -maxdepth 1 -type d \
            \( -name 'weekly-*' -o -name 'release-*' \) \
            -mtime +$THRESHOLD_DAYS -print0 \
          | xargs -0 -r rm -rf
    done <<<"$CLEANUP_PATHS"
products:
  - test_case: [cleanup_old_artifacts]
    products:
      - scope: [cleanup]
        platforms: [dgx_a100, dgx_h100, dgx_gb200]

Test plan

• Trigger weekly-scope pipeline; verify functional:cleanup_dgx_* runs before functional:run_* on the matching cluster.
• Each cleanup job log lists deletions of weekly-* / release-* directories older than 28 days under each base dir, and nothing else.
• mr-scope pipeline does not run cleanup jobs (gated by scope rule).
• Post-run, sanity-check ls on each base dir and confirm only fresh weekly-* / release-* entries remain; all other siblings (e.g. mcore_ci) intact.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[ko3n1g]

/ok to test ffedb81

[ko3n1g]

/ok to test 80c4ed3