Skip to content

Add flexible controls of GPU configuration and remove old GPU configuration#97

Merged
jedwards4b merged 27 commits into
ESMCI:mainfrom
sjsprecious:add_gpu_gust
Aug 9, 2023
Merged

Add flexible controls of GPU configuration and remove old GPU configuration#97
jedwards4b merged 27 commits into
ESMCI:mainfrom
sjsprecious:add_gpu_gust

Conversation

@sjsprecious
Copy link
Copy Markdown
Collaborator

@sjsprecious sjsprecious commented Apr 17, 2023 •
edited
Loading

This PR:

  • updates the configurations for the GPU offload based on the new XML variables in CMEPS (Add new XML variables for flexible controls of GPU configuration ESCOMP/CMEPS#363).
  • removes the old approaches for GPU configurations.
  • updates the module versions and batch script template on Derecho.
  • removes #PBS -V option as it incorrectly populate the NGPUS env variable from the login node to the compute node on Derecho.

@sjsprecious
Add Jim's changes for Gust's modules from jedwards4b@87b9539
9beb292 
Use nvhpc/22.11, cray-mpich/8.1.21, and esmf/8.4.1.b02

	modified:   machines/cmake_macros/gust.cmake
	modified:   machines/config_machines.xml
@sjsprecious
Add Jim's gpu_improvement branch at https://github.com/jedwards4b/ccs…
4134af3 
…_config_cesm/compare/52c06b3..fbc05d6

Update settings on Gust and Casper

	modified:   Depends.nvhpc
	deleted:    Depends.nvhpc-gpu
	deleted:    cmake_macros/nvhpc-gpu.cmake
	deleted:    cmake_macros/nvhpc-gpu_casper.cmake
	modified:   cmake_macros/nvhpc.cmake
	modified:   cmake_macros/nvhpc_casper.cmake
	deleted:    cmake_macros/pgi-gpu.cmake
	deleted:    cmake_macros/pgi-gpu_casper.cmake
	modified:   config_batch.xml
	modified:   config_machines.xml
	deleted:    mpi_run_gpu.casper
@sjsprecious
Add max_cputasks_per_gpu_node, gpu_type and gpu_offload options for C…
3b7a182 
…asper and Gust

	modified:   machines/config_machines.xml
@sjsprecious
update batch script template for Gust GPU node
ee726b5 
	modified:   config_batch.xml
@sjsprecious
typo fix for mismatched tags for GPU options
1f5eb30 
	modified:   machines/config_machines.xml
@sjsprecious
Add missing "MAX_GPUS_PER_NODE"
8649a95 
	modified:   machines/config_machines.xml
@sjsprecious
downgrade netcdf-mpi fpr nvhpc/22.11
e7bfbe1 
	modified:   machines/config_machines.xml
@sjsprecious
bug fix of batch script template with gpu_enabled option
bf14967 
	modified:   machines/config_batch.xml
@sjsprecious
remove -target=zen3 flag on gust
14f605e 
load cuda module on Gust for a GPU run

	deleted:    cmake_macros/nvhpc_gust.cmake
	modified:   config_machines.xml
@sjsprecious
update to nvhpc/23.1 and related modules
59086af 
	modified:   machines/config_machines.xml
@sjsprecious
turn off MPS on Gust
918e3a8 
	modified:   config_batch.xml
@sjsprecious
Update module versions for intel compiler
d0093a5 
	modified:   machines/config_batch.xml
	modified:   machines/config_machines.xml
@sjsprecious
Merge branch 'main' into add_gpu_gust
4023d67
@sjsprecious sjsprecious requested a review from jedwards4b April 17, 2023 16:00
@sjsprecious sjsprecious added the enhancement New feature or request label Apr 17, 2023
@sjsprecious sjsprecious mentioned this pull request Apr 17, 2023
Merged
@sjsprecious
load ncarenv and ncarcompilers manually
7224a37 
	modified:   config_machines.xml
@sjsprecious
update the ncarenv and cesmdev module on Gust
822f7b7 
	modified:   machines/config_machines.xml
@sjsprecious
Copy link
Copy Markdown
Collaborator Author

sjsprecious commented May 26, 2023

@jedwards4b will issue a separate PR to introduce these changes.

@sjsprecious sjsprecious reopened this Aug 8, 2023
@sjsprecious sjsprecious mentioned this pull request Aug 8, 2023
Merged
@sjsprecious sjsprecious requested a review from jedwards4b August 8, 2023 16:33
@sjsprecious
Copy link
Copy Markdown
Collaborator Author

sjsprecious commented Aug 9, 2023

I updated the nvhpc compiler version to 23.5 with a newer version of ESMF8.6.0b03. The GPU run finished successfully on Derecho.

@jedwards4b jedwards4b merged commit fe65a9c into ESMCI:main Aug 9, 2023
@sjsprecious sjsprecious deleted the add_gpu_gust branch August 9, 2023 19:24
@sjsprecious sjsprecious mentioned this pull request Sep 18, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jedwards4b jedwards4b jedwards4b approved these changes

Assignees

@jedwards4b jedwards4b

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sjsprecious @jedwards4b