Skip to content

Remove pslot dir from local archive folder on CI case completion #2961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
TerrenceMcGuinness-NOAA wants to merge 36 commits into from
Closed

Remove pslot dir from local archive folder on CI case completion #2961

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
1c159d8
added a cleanup_experment funciton to ci_utils that cleans local arch…
TerrenceMcGuinness-NOAA Sep 24, 2024
e7aa3ba
Merge branch 'NOAA-EMC:develop' into ci_clean_loc_arch
TerrenceMcGuinness-NOAA Sep 25, 2024
463655a
moved rm for EXPDIR into cleanup_expermiment function
TerrenceMcGuinness-NOAA Sep 25, 2024
b511367
Merge branch 'ci_clean_loc_arch' of github.com:TerrenceMcGuinness-NOA…
TerrenceMcGuinness-NOAA Sep 25, 2024
40c96b2
fixed bracket sytax error in ci_utils and missnamed function call in …
TerrenceMcGuinness-NOAA Sep 26, 2024
54ce216
fixed SC2155 in bash utils for cleanup
TerrenceMcGuinness-NOAA Sep 26, 2024
cacde8c
fixed SC2115 for var expation in path for clean experiment util
TerrenceMcGuinness-NOAA Sep 26, 2024
c8a10a9
needed the OR truch for var assigment via actions in bash
TerrenceMcGuinness-NOAA Sep 26, 2024
7c93f4b
fixed the evaluation of HOMDIR so that internal strification of env v…
TerrenceMcGuinness-NOAA Sep 27, 2024
bb92a39
Merge branch 'develop' into ci_clean_loc_arch
TerrenceMcGuinness-NOAA Sep 27, 2024
bfcbba5
still needed archive tagged back onto HOMEDIR
TerrenceMcGuinness-NOAA Sep 27, 2024
69fc51b
Merge branch 'ci_clean_loc_arch' of github.com:TerrenceMcGuinness-NOA…
TerrenceMcGuinness-NOAA Sep 27, 2024
49b4489
new linee at aend of ci_utils
TerrenceMcGuinness-NOAA Sep 27, 2024
b776a59
Merge branch 'NOAA-EMC:develop' into ci_clean_loc_arch
TerrenceMcGuinness-NOAA Oct 23, 2024
d8af194
removed eval in cleanup_experiment and added envsubst at end of pipe …
TerrenceMcGuinness-NOAA Oct 23, 2024
d54bd8d
updated archival directory to ARCDIR from config.base in cleanup_expe…
TerrenceMcGuinness-NOAA Oct 24, 2024
85f6369
forgot to clean up EXPDIR replacement in cleanup_experment
TerrenceMcGuinness-NOAA Oct 24, 2024
6a09b36
forgot to clean up EXPDIR replacement in cleanup_experment still
TerrenceMcGuinness-NOAA Oct 24, 2024
32b8cd0
Update ci/scripts/utils/ci_utils.sh
TerrenceMcGuinness-NOAA Oct 25, 2024
e01d9d3
added get_config_var util to simplify cleanup_experment
TerrenceMcGuinness-NOAA Nov 5, 2024
3d5e851
removed debug outputs in cleanup_experiment
TerrenceMcGuinness-NOAA Nov 5, 2024
162474a
added next step to use get_config_var to remove previous DATAROOT dat…
TerrenceMcGuinness-NOAA Nov 5, 2024
7da3721
fix conficts and merged develop back in
TerrenceMcGuinness-NOAA Nov 5, 2024
c6ad4ec
updated Jenkinsfile to use clean experiment and to also clean DATADIR…
TerrenceMcGuinness-NOAA Nov 5, 2024
bef32e7
updating correct ATARDIR variable in cleaup
TerrenceMcGuinness-NOAA Nov 5, 2024
02642fc
added true for shell norms
TerrenceMcGuinness-NOAA Nov 5, 2024
62ec3f6
Update ci/scripts/utils/ci_utils.sh
TerrenceMcGuinness-NOAA Nov 6, 2024
7e5b4fb
Update ci/scripts/utils/ci_utils.sh
TerrenceMcGuinness-NOAA Nov 6, 2024
bf3b80f
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
cd4e27b
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
5ea8d3d
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
99b9322
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
f56be33
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
9a381b4
fixed name of get_config_var in pipeline (left off the py extention)
TerrenceMcGuinness-NOAA Nov 6, 2024
788dd8f
DATAROOT is not a config.base value but is construced from STMP in ta…
TerrenceMcGuinness-NOAA Nov 6, 2024
7d3b8bf
updated Jenkins file with correct RUNDIRS string (had RUNDIR as a bug)
TerrenceMcGuinness-NOAA Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ pipeline {
sh(script: " rm -f ${error_file}")
try {
sh(script: "${HOMEgfs}/ci/scripts/run-check_ci.sh ${CUSTOM_WORKSPACE} ${pslot} ${system}")
sh(script: "${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh cleanup_experiment ${CUSTOM_WORKSPACE}/RUNTESTS/EXPDIR/${pslot}")
} catch (Exception error_experment) {
sh(script: "${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh cancel_batch_jobs ${pslot}")
ws(CUSTOM_WORKSPACE) {
Expand Down
3 changes: 1 addition & 2 deletions ci/scripts/check_ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,7 @@ for pr in ${pr_list}; do
fi
if [[ "${rocoto_state}" == "DONE" ]]; then
#Remove Experment cases that completed successfully
rm -Rf "${pslot_dir}"
rm -Rf "${pr_dir}/RUNTESTS/COMROOT/${pslot}"
"${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh" cleanup_experiment "${pslot_dir}"
rm -f "${output_ci_single}"
# echo "\`\`\`" > "${output_ci_single}"
DATE=$(date +'%D %r')
Expand Down
22 changes: 22 additions & 0 deletions ci/scripts/utils/ci_utils.sh
View file
Open in desktop
Comment thread
DavidHuber-NOAA marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,25 @@ function publish_logs() {
fi
echo "${URL}"
}

function cleanup_experiment() {
Comment thread Fixed
Comment thread Fixed
# cleanup_experiment function
# This function takes a directory path of an EXPDIR and
# gets HOMEDIR value from config.base to
# clean up the archived files on disk.

local PSLOT_PATH
local pslot
local HOMEDIR
local ARCHIVEDIR

PSLOT_PATH="$1"
Comment thread Fixed
pslot=$(basename "${PSLOT_PATH}")
HOMEDIR=$(grep 'export HOMEDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]')
Comment thread Fixed
Comment thread Fixed
ARCHIVEDIR="${HOMEDIR}/archive"

@WalterKolczynski-NOAA WalterKolczynski-NOAA Oct 23, 2024 •
edited by TerrenceMcGuinness-NOAA
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be reading in $ARCDIR from config.base, similar to how you are reading in $HOMEDIR. Note that the default ARCDIR is ${NOSCRUB}/archive, not ${HOMEDIR}/archive. $ARCDIR already includes the pslot, so it won't need to be added to the next line.

@WalterKolczynski-NOAA On the other hand export NOSCRUB="$HOMEDIR" is also in confg.base

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Oct 24, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we should have lead with that.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Oct 24, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok updated using ARCDIR.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Oct 28, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test failed because ARCDIR has an unresolved sub-string ${NOSCRUB} derived from ${HOMEDIR}

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Oct 31, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WalterKolczynski-NOAA David Huber came up with idea of scraping the archive log files in COMROOT:

mterry (hercules-login-4) COMROOT $ echo $COMROOT
C96_S2SWA_gefs_replay_ics_32b8cd0c
mterry (hercules-login-4) COMROOT $ sed -n "s/file_utils  : Copied [[:graph:]]* to \([[:graph:]]*\)/\1/p" "${COMROOT}"/logs/*/*arch.log | awk '{print $NF}' | uniq
/work2/noaa/stmp/CI/HERCULES/2961/RUNTESTS/COMROOT/C96_S2SWA_gefs_replay_ics_32b8cd0c/gefsarch

mterry (hercules-login-4) COMROOT $ ls /work2/noaa/stmp/CI/HERCULES/2961/RUNTESTS/COMROOT/C96_S2SWA_gefs_replay_ics_32b8cd0c/gefsarch
gefs.t00z.mean.pres_.1p00.f006.grib2  gefs.t00z.mean.pres_.1p00.f012.grib2  gefs.t00z.mean.pres_.1p00.f018.grib2  gefs.t00z.mean.pres_.1p00.f024.grib2	gefs.t00z.mean.pres_.1p00.f030.grib2  gefs.t00z.mean.pres_.1p00.f036.grib2  gefs.t00z.mean.pres_.1p00.f042.grib2  gefs.t00z.mean.pres_.1p00.f048.grib2

I'll get that in tomorrow as another proposed solution.

@WalterKolczynski-NOAA WalterKolczynski-NOAA Nov 4, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of that, why not just feed the value of $NOSCRUB in?

ARCHIVEDIR=$(grep 'export ARCDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]' | NOSCRUB="${HOMEDIR}" envsubst || true) || true

Or if we want to be more rigorous

pslot=$(basename "${PSLOT_PATH}")
HOMEDIR=$(grep 'export HOMEDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]' | envsubst || true) || true
NOSCRUB=$(grep 'export NOSCRUB=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]' | HOMEDIR="${HOMEDIR}" envsubst || true) || true
ARCHIVEDIR=$(grep 'export ARCDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]' | NOSCRUB="${NOSCRUB}" envsubst || true) || true

Or we could just cut to the chase and source config.base and get everything.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Nov 5, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO "reconstructing" paths from hidden knowledge of their specificity such as interim archive placements is a non-systematic and haphazard approach resulting in a one-off paradigm. If we really want to be rigorous I would propose a transformation from logs to a systematic and intent-full pedigree framework that could be leveraged for a requirement such as this.


rm -rf "${ARCHIVEDIR}/${pslot}"
Comment thread Fixed
rm -Rf "${PSLOT_PATH}"
rm -Rf "${PSLOT_PATH}/../COMROOT/${pslot}"
}

Comment thread Fixed
Comment thread Fixed