Ci enable distconv #2235
Merged
Ci enable distconv #2235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bvanessen
force-pushed
the
ci_enable_distconv
branch
from
d1860a9 to
752e1c6
Compare
tbennun
reviewed
Mar 26, 2023
tbennun
approved these changes
Mar 29, 2023
bvanessen
force-pushed
the
ci_enable_distconv
branch
from
752e1c6 to
7d29a79
Compare
Added DistConv CI tests Added Corona DistConv test and disabled FFT on ROCm Ensure that DistConv tests keep error signals Enable NVSHMEM on Lassen Added a multi-stage pipeline for Lassen Fixed a typo and disabled other tests. Added spack environment Added check stage for the catch tests Debugging Added the definition of the RESULTS_DIR environment variable Added release notes. Fixed the launcher for catch tests Removed debugging code Changed the batch launch commands to be interactive to block completion. Added a wrapper shell script for launching the unit tests Added the number of nodes for the unit test. Cleaning up launching paths Added execute permissions for unit test script. Ingest the Spack dependent environment information. Fixed typo. Disabled the external test on Lassen with NVSHMEM and DistConv. Fixing launch command and exclusion of externallayer Bugfix python Added number of tasks per node Added integration tests. Set some NVSHMEM runtime variables Cleaned up debugging code Fixed typos Run the correct test Restore the CI testing Uniquify the CI JOB_NAME fields for DistConv tests.
common python tools. Switched all tests to using the standard contrib args version.
testing, both for normal users and lbannusr.
benson31
previously requested changes
Sep 21, 2023
.gitlab-ci.yml
Outdated
Comment on lines
65
to
76
| # lassen distconv testing: | ||
| # stage: run-all-clusters | ||
| # variables: | ||
| # JOB_NAME_SUFFIX: _distconv | ||
| # SPACK_ENV_BASE_NAME_MODIFIER: "-multi-stage-distconv" | ||
| # SPACK_SPECS: "+cuda +distconv +nvshmem +fft" | ||
| # WITH_WEEKLY: "${LBANN_CI_RUN_WEEKLY}" | ||
| # WITH_CLEAN_BUILD: "${LBANN_CI_CLEAN_BUILD}" | ||
| # TEST_FLAG: "test_*_distconv.py" | ||
| # trigger: | ||
| # strategy: depend | ||
| # include: .gitlab/lassen/multi_stage_pipeline.yml |
There was a problem hiding this comment.
What's the issue here? Arguably this is the most important of the test jobs for this PR...
benson31
reviewed
Sep 21, 2023
erronously resetting up the spack environment. Changed the saved spack environment name to SPACK_ENV_NAME. Cleaned up some dead code.
always at least one true clause, so the stage will schedule. Fixed the regex so that the distconv substring doesn't have to come at the start of the string.
Co-authored-by: Tom Benson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added new CI tests for building and checking execution with both DistConv and NVSHMEM enabled as appropriate.