Skip to content

update workflow to use actions from cdeps #332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jedwards4b merged 17 commits into from
Dec 27, 2022
Merged

update workflow to use actions from cdeps #332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
4e32aed
update workflow to use actions from cdeps
jedwards4b Dec 23, 2022
7590341
needs full SHA
jedwards4b Dec 23, 2022
64f71d7
need to setup environment
jedwards4b Dec 23, 2022
238b861
add scripts_regression_tests to workflow
jedwards4b Dec 23, 2022
d53f965
add scripts_regression_tests to workflow
jedwards4b Dec 23, 2022
6f76ccc
ref not sha
jedwards4b Dec 23, 2022
fa2ccbf
nether ref nor sha
jedwards4b Dec 23, 2022
9bba7a3
fix path
jedwards4b Dec 23, 2022
b23d443
fix cmeps path
jedwards4b Dec 23, 2022
caba810
fix cmeps path
jedwards4b Dec 23, 2022
839d8e9
fix cmeps path
jedwards4b Dec 23, 2022
f965da9
add cpl7
jedwards4b Dec 23, 2022
2fd947b
install PyYAML
jedwards4b Dec 23, 2022
f2f06c2
turn on debug
jedwards4b Dec 27, 2022
0dde540
use pio external
jedwards4b Dec 27, 2022
cd6e6e2
set more env variables
jedwards4b Dec 27, 2022
281332b
try adding pio
jedwards4b Dec 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 45 additions & 71 deletions .github/workflows/extbuild.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,103 +19,77 @@ jobs:
CXX: mpicxx
CPPFLAGS: "-I/usr/include -I/usr/local/include"
# Versions of all dependencies can be updated here
ESMF_VERSION: v8.3.0b13
PNETCDF_VERSION: pnetcdf-1.12.3
NETCDF_FORTRAN_VERSION: v4.5.2
PIO_VERSION: pio2_5_7
ESMF_VERSION: v8.4.0
PNETCDF_VERSION: checkpoint.1.12.3
NETCDF_FORTRAN_VERSION: v4.6.0
PIO_VERSION: pio2_5_10
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
# Build the ESMF library, if the cache contains a previous build
# it will be used instead
- id: cache-esmf
uses: actions/cache@v2
with:
path: ~/ESMF
key: ${{ runner.os }}-${{ env.ESMF_VERSION }}-ESMF
- id: load-env
run: |
sudo apt-get update
sudo apt-get install gfortran wget openmpi-bin netcdf-bin libopenmpi-dev libnetcdf-dev
- id: checkout-ESMF
uses: actions/checkout@v3
sudo apt-get install gfortran wget openmpi-bin netcdf-bin libopenmpi-dev libnetcdf-dev autotools-dev autoconf
- id: cache-esmf
uses: actions/cache@v3
with:
repository: esmf-org/esmf
path: esmf-src
ref: ${{ env.ESMF_VERSION }}
- id: build-ESMF
if: steps.cache-esmf.outputs.cache-hit != 'true'
run: |
#wget https://github.com/esmf-org/esmf/archive/${{ env.ESMF_VERSION }}.tar.gz
#tar -xzvf ${{ env.ESMF_VERSION }}.tar.gz
#pushd esmf-${{ env.ESMF_VERSION }}
pushd esmf-src
export ESMF_DIR=`pwd`
export ESMF_COMM=openmpi
export ESMF_YAMLCPP="internal"
export ESMF_INSTALL_PREFIX=$HOME/ESMF
export ESMF_BOPT=g
make
make install
popd
path: ~/ESMF
key: ${{ runner.os }}-${{ env.ESMF_VERSION }}-ESMF
- id: cache-pnetcdf
uses: actions/cache@v2
with:
path: ~/pnetcdf
key: ${{ runner.os }}-${{ env.PNETCDF_VERSION}}-pnetcdf
- name: pnetcdf build
if: steps.cache-pnetcdf.outputs.cache-hit != 'true'
run: |
wget https://parallel-netcdf.github.io/Release/${{ env.PNETCDF_VERSION }}.tar.gz
tar -xzvf ${{ env.PNETCDF_VERSION }}.tar.gz
ls -l
pushd ${{ env.PNETCDF_VERSION }}
./configure --prefix=$HOME/pnetcdf --enable-shared --disable-cxx
make
make install
popd
- name: Cache netcdf-fortran
id: cache-netcdf-fortran
uses: actions/cache@v2
with:
path: ~/netcdf-fortran
key: ${{ runner.os }}-${{ env.NETCDF_FORTRAN_VERSION }}-netcdf-fortran
- name: netcdf fortran build
if: steps.cache-netcdf-fortran.outputs.cache-hit != 'true'
run: |
wget https://github.com/Unidata/netcdf-fortran/archive/${{ env.NETCDF_FORTRAN_VERSION }}.tar.gz
tar -xzvf ${{ env.NETCDF_FORTRAN_VERSION }}.tar.gz
ls -l
pushd netcdf-fortran-*
./configure --prefix=$HOME/netcdf-fortran
make
make install

- name: Cache PIO
id: cache-PIO
- name: Cache ParallelIO
id: cache-ParallelIO
uses: actions/cache@v2
with:
path: ~/pio
key: ${{ runner.os }}-${{ env.PIO_VERSION }}.pio
restore-keys: |
${{ runner.os }}-${{ env.NETCDF_FORTRAN_VERSION }}-netcdf-fortran
${{ runner.os }}-${{ env.PNETCDF_VERSION }}-pnetcdf

- id: checkout-PIO
uses: actions/checkout@v3
- name: Build PNetCDF
if: steps.cache-pnetcdf.outputs.cache-hit != 'true'
uses: ESCOMP/CDEPS/.github/actions/buildpnetcdf@e06246b560d3132170bb1a5443fa3d65dfbd2040
with:
repository: NCAR/ParallelIO
path: parallelio-src
ref: ${{ env.PIO_VERSION }}
- name: Build PIO
if: steps.cache-PIO.outputs.cache-hit != 'true'
run: |
mkdir build-pio
pushd build-pio
cmake -Wno-dev -DNetCDF_C_LIBRARY=/usr/lib/x86_64-linux-gnu/libnetcdf.so -DNetCDF_C_INCLUDE_DIR=/usr/include -DCMAKE_PREFIX_PATH=/usr -DCMAKE_INSTALL_PREFIX=$HOME/pio -DPIO_HDF5_LOGGING=On -DPIO_USE_MALLOC=On -DPIO_ENABLE_TESTS=Off -DPIO_ENABLE_LOGGING=On -DPIO_ENABLE_EXAMPLES=Off -DPIO_ENABLE_TIMING=Off -DNetCDF_Fortran_PATH=$HOME/netcdf-fortran -DPnetCDF_PATH=$HOME/pnetcdf ../parallelio-src
make VERBOSE=1
make install
popd

pnetcdf_version: ${{ env.PNETCDF_VERSION }}
install_prefix: $HOME/pnetcdf
- name: Build NetCDF Fortran
if: steps.cache-netcdf-fortran.outputs.cache-hit != 'true'
uses: ESCOMP/CDEPS/.github/actions/buildnetcdff@e06246b560d3132170bb1a5443fa3d65dfbd2040
with:
netcdf_fortran_version: ${{ env.NETCDF_FORTRAN_VERSION }}
install_prefix: $HOME/netcdf-fortran
netcdf_c_path: /usr
- name: Build ParallelIO
if: steps.cache-ParallelIO.outputs.cache-hit != 'true'
uses: ESCOMP/CDEPS/.github/actions/buildpio@e06246b560d3132170bb1a5443fa3d65dfbd2040
with:
parallelio_version: ${{ env.ParallelIO_VERSION }}
netcdf_c_path: /usr
netcdf_fortran_path: $HOME/netcdf-fortran
pnetcdf_path: $HOME/pnetcdf
install_prefix: $HOME/pio
- name: Build ESMF
if: steps.cache-esmf.outputs.cache-hit != 'true'
uses: ESCOMP/CDEPS/.github/actions/buildesmf@e06246b560d3132170bb1a5443fa3d65dfbd2040
with:
esmf_version: ${{ env.ESMF_VERSION }}
esmf_bopt: g
esmf_comm: openmpi
install_prefix: $HOME/ESMF
netcdf_c_path: /usr
netcdf_fortran_path: $HOME/netcdf-fortran
pnetcdf_path: $HOME/pnetcdf
parallelio_path: $HOME/pio
- name: Build CMEPS
run: |
export ESMFMKFILE=$HOME/ESMF/lib/libg/Linux.gfortran.64.openmpi.default/esmf.mk
Expand Down
175 changes: 91 additions & 84 deletions .github/workflows/srt.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ name: scripts regression tests
# events but only for the master branch
on:
push:
branches: main
branches: [ master ]
pull_request:
branches: main
branches: [ master ]

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
Expand All @@ -18,132 +18,139 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8, 3.9]
python-version: [ 3.10.9 ]
env:
CC: mpicc
FC: mpifort
CXX: mpicxx
CPPFLAGS: "-I/usr/include -I/usr/local/include"
# Versions of all dependencies can be updated here
PNETCDF_VERSION: pnetcdf-1.12.2
NETCDF_FORTRAN_VERSION: v4.5.2
MCT_VERSION: MCT_2.11.0
PARALLELIO_VERSION: pio2_5_4
PNETCDF_VERSION: checkpoint.1.12.3
NETCDF_FORTRAN_VERSION: v4.6.0
ESMF_VERSION: v8.4.0
PARALLELIO_VERSION: pio2_5_10
NETCDF_C_PATH: /usr
NETCDF_FORTRAN_PATH: ${HOME}/netcdf-fortran
PNETCDF_PATH: ${HOME}/pnetcdf
CIME_MODEL: cesm
CIME_DRIVER: mct

CIME_DRIVER: nuopc
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it

- name: cime checkout
uses: actions/checkout@v2
with:
repository: ESMCI/cime

- name: share checkout
uses: actions/checkout@v2
with:
repository: ESCOMP/CESM_share
path: share

- name: cpl7 checkout
uses: actions/checkout@v2
with:
repository: ESCOMP/CESM_CPL7andDataComps
path: components/cpl7

- id: load-env
run: |
sudo apt-get update
sudo apt-get install libxml2-utils pylint wget gfortran openmpi-bin netcdf-bin libopenmpi-dev cmake libnetcdf-dev
sudo apt-get install libxml2-utils pylint wget gfortran openmpi-bin netcdf-bin libopenmpi-dev cmake libnetcdf-dev autotools-dev autoconf

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: mct install
run: |
git clone -b ${{ env.MCT_VERSION }} https://github.com/MCSclimate/MCT libraries/mct
ls -l libraries/mct

- name: parallelio install
- run: echo "PyYAML" > requirements.txt
- name: Install PyYAML
run: pip install -r requirements.txt
# use the latest cesm master
- name: cesm checkout
uses: actions/checkout@v3
with:
repository: ESCOMP/CESM
path: cesm
# this cmeps commit
- name: cmeps checkout
uses: actions/checkout@v3
with:
path: cesm/components/cmeps

# Checkout cesm datamodels and support
# cpl7 is needed - i think that's a bug
- name: checkout externals
run: |
git clone -b ${{ env.PARALLELIO_VERSION }} https://github.com/NCAR/ParallelIO libraries/parallelio
ls -l libraries/parallelio

pushd cesm
./manage_externals/checkout_externals ccs_config cdeps cime share mct cpl7 parallelio

- name: Cache ESMF
id: cache-esmf
uses: actions/cache@v3
with:
path: ~/ESMF
key: ${{ runner.os }}-${{ env.ESMF_VERSION }}-ESMF
- name: cache pnetcdf
id: cache-pnetcdf
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/pnetcdf
key: ${{ runner.os }}-${{ env.PNETCDF_VERSION}}-pnetcdf-redo

- name: pnetcdf build
if: steps.cache-pnetcdf.outputs.cache-hit != 'true'
run: |
wget https://parallel-netcdf.github.io/Release/${{ env.PNETCDF_VERSION }}.tar.gz
tar -xzvf ${{ env.PNETCDF_VERSION }}.tar.gz
ls -l
pushd ${{ env.PNETCDF_VERSION }}
./configure --prefix=$HOME/pnetcdf --enable-shared --disable-cxx
make
make install
popd
key: ${{ runner.os }}-${{ env.PNETCDF_VERSION}}-pnetcdf

- name: Cache netcdf-fortran
id: cache-netcdf-fortran
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/netcdf-fortran
key: ${{ runner.os }}-${{ env.NETCDF_FORTRAN_VERSION }}-netcdf-fortran-redo

- name: netcdf fortran build
if: steps.cache-netcdf-fortran.outputs.cache-hit != 'true'
run: |
sudo apt-get install libnetcdf-dev
wget https://github.com/Unidata/netcdf-fortran/archive/${{ env.NETCDF_FORTRAN_VERSION }}.tar.gz
tar -xzvf ${{ env.NETCDF_FORTRAN_VERSION }}.tar.gz
ls -l
pushd netcdf-fortran-*
./configure --prefix=$HOME/netcdf-fortran
make
make install

- name: link netcdf-c to netcdf-fortran path
# link netcdf c library here to simplify build
run: |
pushd ${{ env.NETCDF_FORTRAN_PATH }}/include
ln -fs /usr/include/*netcdf* .
pushd ${{ env.NETCDF_FORTRAN_PATH }}/lib
clibdir=`nc-config --libdir`
ln -fs $clibdir/lib* .
key: ${{ runner.os }}-${{ env.NETCDF_FORTRAN_VERSION }}-netcdf-fortran

- name: Cache ParallelIO
id: cache-ParallelIO
uses: actions/cache@v3
with:
path: ~/pio
key: ${{ runner.os }}-${{ env.PARALLELIO_VERSION }}.pio
- name: Cache inputdata
id: cache-inputdata
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: $HOME/cesm/inputdata
key: inputdata
#
# The following can be used to ssh to the testnode for debugging
# see https://github.com/mxschmitt/action-tmate for details
# - name: Setup tmate session
# uses: mxschmitt/action-tmate@v3
- name: Build PNetCDF
if: steps.cache-pnetcdf.outputs.cache-hit != 'true'
uses: ESCOMP/CDEPS/.github/actions/buildpnetcdf@e06246b560d3132170bb1a5443fa3d65dfbd2040
with:
pnetcdf_version: ${{ env.PNETCDF_VERSION }}
install_prefix: $HOME/pnetcdf
- name: Build NetCDF Fortran
if: steps.cache-netcdf-fortran.outputs.cache-hit != 'true'
uses: ESCOMP/CDEPS/.github/actions/buildnetcdff@e06246b560d3132170bb1a5443fa3d65dfbd2040
with:
netcdf_fortran_version: ${{ env.NETCDF_FORTRAN_VERSION }}
install_prefix: $HOME/netcdf-fortran
netcdf_c_path: /usr
- name: Build ParallelIO
if: steps.cache-PARALLELIO.outputs.cache-hit != 'true'
uses: ESCOMP/CDEPS/.github/actions/buildpio@e06246b560d3132170bb1a5443fa3d65dfbd2040
with:
parallelio_version: ${{ env.ParallelIO_VERSION }}
netcdf_c_path: /usr
netcdf_fortran_path: $HOME/netcdf-fortran
pnetcdf_path: $HOME/pnetcdf
install_prefix: $HOME/pio
- name: Build ESMF
if: steps.cache-esmf.outputs.cache-hit != 'true'
uses: ESCOMP/CDEPS/.github/actions/buildesmf@e06246b560d3132170bb1a5443fa3d65dfbd2040
with:
esmf_version: ${{ env.ESMF_VERSION }}
esmf_bopt: g
esmf_comm: openmpi
install_prefix: $HOME/ESMF
netcdf_c_path: /usr
netcdf_fortran_path: $HOME/netcdf-fortran
pnetcdf_path: $HOME/pnetcdf
parallelio_path: $HOME/pio

- name: scripts regression tests
run: |
mkdir -p $HOME/cesm/scratch
mkdir -p $HOME/cesm/inputdata
cd $HOME/work/CESM_share/CESM_share/scripts/tests
ls -l $HOME/work/CESM_share/CESM_share
cd $GITHUB_WORKSPACE/cesm/cime/CIME/tests
export CIME_TEST_PLATFORM=ubuntu-latest
export PIO_INCDIR=$HOME/pio/include
export PIO_LIBDIR=$HOME/pio/lib
export PIO_VERSION_MAJOR=2
export PIO_TYPENAME_VALID_VALUES="netcdf,pnetcdf,netcdf4p,netcdf4c"
export NETCDF=$HOME/netcdf-fortran
export PATH=$NETCDF/bin:$PATH
export PATH=$NETCDF/bin:$PATH:$HOME/netcdf-fortran/bin
export LD_LIBRARY_PATH=$NETCDF/lib:$HOME/pnetcdf/lib:$LD_LIBRARY_PATH
export ESMFMKFILE=$HOME/ESMF/lib/libg/Linux.gfortran.64.openmpi.default/esmf.mk
./scripts_regression_tests.py --no-fortran-run --compiler gnu --mpilib openmpi --machine ubuntu-latest

# the following can be used by developers to login to the github server in case of errors
Expand Down