Skip to content

Update ufs-bundle to build with head of develop of ufs-weather-model for ATM configuration #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
climbfuji merged 49 commits into from
Feb 2, 2024
Merged

Update ufs-bundle to build with head of develop of ufs-weather-model for ATM configuration #43

Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
db104fe
working version with calculation of ensemble mean values
mark-a-potts Oct 20, 2023
f6b30cd
pegged hashes for ioda, saber, ufo
mark-a-potts Oct 20, 2023
e307be2
updated fv3-jedi-data branch
mark-a-potts Oct 20, 2023
7ad6bc6
Updated to calculate the ensemble mean of the StateSet
mark-a-potts Nov 22, 2023
3119a2c
Updating to head of develop for ufs-wm
mark-a-potts Nov 29, 2023
0dd0a61
Added script to regenerate rundir for c48 case
mark-a-potts Nov 29, 2023
7b6c402
updates to handle head of develop for ufs-wm
mark-a-potts Nov 30, 2023
30f7e45
different wm repos for non-ATM builds
mark-a-potts Dec 5, 2023
d55e8f7
updated to build s2s and aero
mark-a-potts Dec 5, 2023
1bf0d74
slight changes to README
mark-a-potts Dec 5, 2023
0853610
added patch for checksum checking
mark-a-potts Dec 5, 2023
cedd860
Merge remote-tracking branch 'origin/develop' into feature/ufs-EnsFor…
mark-a-potts Dec 5, 2023
6263d82
cleanup
mark-a-potts Dec 5, 2023
2965dae
Updates for building ATM, ATMAERO, and S2S
mark-a-potts Dec 7, 2023
0e9437e
Cleaned up CMakeLists.txt a bit
mark-a-potts Dec 7, 2023
dd72818
Merge branch 'develop' into feature/ufs-EnsForecasts
climbfuji Dec 7, 2023
4004a52
removed extra jedi-data line
mark-a-potts Dec 7, 2023
272e8b3
Merge branch 'develop' into feature/ufs-EnsForecasts
climbfuji Dec 19, 2023
fbf67ae
cleanup
mark-a-potts Dec 20, 2023
dd406b5
Merge branch 'feature/ufs-EnsForecasts' of https://github.com/jcsda/u…
mark-a-potts Dec 20, 2023
6e031eb
updated github workflows
mark-a-potts Dec 20, 2023
5fdfb94
added verbose to test
mark-a-potts Dec 21, 2023
207abee
added verbose to test
mark-a-potts Dec 21, 2023
7da4abd
fixed verbose test
mark-a-potts Dec 21, 2023
eba1898
added space to trigger test
mark-a-potts Dec 21, 2023
7f57d2e
more triggers for ci test
mark-a-potts Dec 21, 2023
ca597c7
removed triggers
mark-a-potts Dec 21, 2023
0cb33ca
added UPDATE for fv3-jedi bundle
mark-a-potts Dec 21, 2023
0b3dd5a
added verbose output for ctests
mark-a-potts Dec 21, 2023
6f0a04b
fixed verbose output for ctests
mark-a-potts Dec 21, 2023
9676841
temporary change for soca compilation
mark-a-potts Jan 2, 2024
fc0ff17
Tweaking tests
mark-a-potts Jan 2, 2024
bdbf774
working on ci tests
mark-a-potts Jan 2, 2024
7ac0a29
more tweaks to ci
mark-a-potts Jan 2, 2024
113f319
Extended patch for wm and changed to official wm repo
mark-a-potts Jan 3, 2024
59a9064
switched oops back to develop
mark-a-potts Jan 5, 2024
618905d
Updates for PR
mark-a-potts Jan 8, 2024
70bec2e
Switched patch to only appy to ng-godas after UFS-WM PR was merged in…
mark-a-potts Jan 12, 2024
70608c6
Fixed patch and updated CMakeLists.txt to account for fv3-jedi pr
mark-a-potts Jan 31, 2024
f5849ec
Merge branch 'develop' into feature/ufs-EnsForecasts
mark-a-potts Feb 1, 2024
e39e115
Updated fms to 2023.02.01 for ufs
mark-a-potts Feb 1, 2024
470f196
tweak to remove ufs-weather-model directory in build directory
mark-a-potts Feb 1, 2024
13c0ee7
tweak to remove everything in the build directory
mark-a-potts Feb 1, 2024
a24ff69
tweak to remove everything in the build-aero directory, re-enable god…
mark-a-potts Feb 1, 2024
393ad39
Update CMakeLists.txt
mark-a-potts Feb 1, 2024
3b78fac
Update CMakeLists.txt
mark-a-potts Feb 1, 2024
36a97c2
Update CMakeLists.txt
mark-a-potts Feb 1, 2024
136543d
Revert changes from README and get rid of rm -rf in yaml
mark-a-potts Feb 1, 2024
d322768
remove build directories
mark-a-potts Feb 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 19 additions & 21 deletions .github/workflows/run_ec2_pcluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ jobs:
module load stack-python/3.10.8

module load jedi-ufs-env/1.0.0
module load fms/2023.02.01

module li

Expand Down Expand Up @@ -135,38 +136,35 @@ jobs:
make -j8

# Runs mandatory ctests on the head node
ctest -R get_
ctest -R fv3jedi_setup_ufs_rundir_warmstart
ctest -R fv3jedi_setup_ufs_rundir_coldstart
ctest -R fv3jedi_test_tier1_model_ufs_warmstart
#ctest -R fv3jedi_test_tier1_model_ufs_coldstart
ctest -R fv3jedi_test_tier1_forecast_ufs_warmstart
#ctest -R fv3jedi_test_tier1_forecast_ufs_coldstart
rm -rf fv3-jedi/test/Data/ModelRunDirs/UFS_warmstart/*
rm -rf fv3-jedi/test/Data/ModelRunDirs/UFS_warmstart_2/*
ctest -R ufs_

# UFS_APP=ATMAERO
cd ${JEDI_ENV}
mkdir -p build-atmaero
cd build-atmaero
rm -rf *

@climbfuji climbfuji Feb 1, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

@mark-a-potts mark-a-potts Feb 1, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to put it there to make sure the build was started fresh. I'll remove it now.

cmake -DUFS_APP=ATMAERO ../ufs-bundle
make update
make -j8

# Skip those for now - linker errors because of static/dynamic library mix
## UFS_APP=NG-GODAS
#cd ${JEDI_ENV}
#mkdir -p build-ng-godas
#cd build-ng-godas
#cmake -DUFS_APP=NG-GODAS ../ufs-bundle
#make update
#make -j8
UFS_APP=NG-GODAS
cd ${JEDI_ENV}
mkdir -p build-ng-godas
cd build-ng-godas
cmake -DUFS_APP=NG-GODAS ../ufs-bundle
make update
make -j8
#
## UFS_APP=S2S
#cd ${JEDI_ENV}
#mkdir -p build-s2s
#cd build-s2s
#cmake -DUFS_APP=S2S ../ufs-bundle
#make update
#make -j8
# UFS_APP=S2S
cd ${JEDI_ENV}
mkdir -p build-s2s
cd build-s2s
cmake -DUFS_APP=S2S ../ufs-bundle
make update
make -j8

- name: Report Status
if: always()
Expand Down
81 changes: 47 additions & 34 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
# ##########

cmake_minimum_required( VERSION 3.12 FATAL_ERROR )
include( ExternalProject )

find_package( ecbuild 3.5 REQUIRED HINTS ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/../ecbuild)

Expand Down Expand Up @@ -41,6 +42,7 @@ find_package(FMS 2022.04 REQUIRED COMPONENTS R4 R8)

# Core JEDI repositories
# ----------------------
#ecbuild_bundle( PROJECT oops GIT "https://github.com/jcsda-internal/oops.git" BRANCH feature/ufs-stateset)
ecbuild_bundle( PROJECT oops GIT "https://github.com/jcsda-internal/oops.git" BRANCH develop UPDATE )
ecbuild_bundle( PROJECT vader GIT "https://github.com/jcsda-internal/vader.git" BRANCH develop UPDATE )
ecbuild_bundle( PROJECT saber GIT "https://github.com/jcsda-internal/saber.git" BRANCH develop UPDATE )
Expand All @@ -58,9 +60,10 @@ ecbuild_bundle( PROJECT ioda-data GIT "https://github.com/JCSDA-internal/ioda-da
ecbuild_bundle( PROJECT ioda GIT "https://github.com/jcsda-internal/ioda.git" BRANCH develop UPDATE )

option(ENABLE_UFO_DATA "Obtain ufo test data from ufo-data repository (vs tarball)" ON)
ecbuild_bundle( PROJECT ufo-data GIT "https://github.com/jcsda-internal/ufo-data.git" BRANCH develop UPDATE )
ecbuild_bundle( PROJECT ufo-data GIT "https://github.com/JCSDA-internal/ufo-data.git" BRANCH develop UPDATE )
Comment thread
mark-a-potts marked this conversation as resolved.
Outdated
ecbuild_bundle( PROJECT ufo GIT "https://github.com/jcsda-internal/ufo.git" BRANCH develop UPDATE )


Comment thread
mark-a-potts marked this conversation as resolved.
Outdated
# Options for building with certain models
# ----------------------------------------
set(UFS_APP "ATM" CACHE STRING "Choose the UFS Application")
Expand All @@ -79,8 +82,8 @@ include_directories(${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_INCLUDEDIR})
include_directories(${DEPEND_LIB_ROOT}/include_r8)
link_directories(${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR})

include( ExternalProject )

# Set variables for UFS clone and build
# Needed to get correct OpenMP link libraries on macOS
set(UFS_CMAKE_EXE_LINKER_FLAGS "-L${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}")
find_package(OpenMP REQUIRED)
Expand All @@ -96,12 +99,23 @@ else()
set(UFS_CMAKE_BUILD_TYPE_FLAG "-DREPRO=ON")
endif()


set(UFS_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/ufs-weather-model)
set(UFS_REPO https://github.com/ufs-community/ufs-weather-model)
set(UFS_TAG "develop" )
if(UFS_APP MATCHES "^(NG-GODAS)$")
set(patch_command ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_CURRENT_SOURCE_DIR}/patches/ng-godas.patch <SOURCE_DIR> && cd <SOURCE_DIR> && git apply ng-godas.patch)
else()
set(patch_command "")
endif()

ExternalProject_Add(ufs-weather-model
PREFIX ${CMAKE_CURRENT_BINARY_DIR}/ufs-weather-model
GIT_REPOSITORY https://github.com/climbfuji/ufs-weather-model
GIT_REPOSITORY ${UFS_REPO}
GIT_SUBMODULES_RECURSE TRUE
GIT_TAG feature/consolidate_dom_update_20221114 # updated from develop on 2023/07/17
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/ufs-weather-model
GIT_TAG ${UFS_TAG}
PATCH_COMMAND ${patch_command}
SOURCE_DIR ${UFS_SOURCE_DIR}
UPDATE_DISCONNECTED ON
INSTALL_DIR ${DEPEND_LIB_ROOT}
# DH* 20230316 turn off OpenMP for now ...
Expand Down Expand Up @@ -133,42 +147,41 @@ add_library( stochastic_physics IMPORTED STATIC)
add_library( fv3atm IMPORTED STATIC)
add_library( ccppphys IMPORTED STATIC)
add_library( mom6 IMPORTED STATIC)
set_target_properties( ufs PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libufs.a)
set_target_properties( ufs_aerosols PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libUFS_Aerosols.a)
set_target_properties( aerosols_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libAerosol_GridComp.a)
set_target_properties( gocart_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libGOCART2G_GridComp.a)
set_target_properties( ca2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libCA2G_GridComp.a)
set_target_properties( chem_shared PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libChem_Shared2G.a)
set_target_properties( gocart_process PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libProcess_Library.a)
set_target_properties( du2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libDU2G_GridComp.a)
set_target_properties( ni2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libNI2G_GridComp.a)
set_target_properties( ss2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libSS2G_GridComp.a)
set_target_properties( su2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libSU2G_GridComp.a)
set_target_properties( ga_environment PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libGA_Environment.a)
set_target_properties( cdeps PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libcdeps.a)
set_target_properties( cmeps PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libcmeps.a)
set_target_properties(ccpp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libccpp_framework.a)
set_target_properties(stochastic_physics PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libstochastic_physics.a)
set_target_properties(fv3atm PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libfv3atm.a)
set_target_properties(ccppphys PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libccpp_physics.a)
set_target_properties(mom6 PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libmom6.a)

if(UFS_APP MATCHES "^(ATM)$" OR UFS_APP MATCHES "^(ATMAERO)$" OR UFS_APP MATCHES "^(S2S)$")
ecbuild_bundle( PROJECT femps GIT "https://github.com/jcsda-internal/femps.git" TAG 1.2.0 )
set_target_properties( ufs PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libufs.a )
set_target_properties( ufs_aerosols PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libUFS_Aerosols.a )
set_target_properties( aerosols_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libAerosol_GridComp.a )
set_target_properties( gocart_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libGOCART2G_GridComp.a )
set_target_properties( ca2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libCA2G_GridComp.a )
set_target_properties( chem_shared PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libChem_Shared2G.a )
set_target_properties( gocart_process PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libProcess_Library.a )
set_target_properties( du2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libDU2G_GridComp.a )
set_target_properties( ni2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libNI2G_GridComp.a )
set_target_properties( ss2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libSS2G_GridComp.a )
set_target_properties( su2g_gridcomp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libSU2G_GridComp.a )
set_target_properties( ga_environment PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libGA_Environment.a )
set_target_properties( cdeps PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libcdeps.a )
set_target_properties( cmeps PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libcmeps.a )
set_target_properties( ccpp PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libccpp_framework.a )
set_target_properties( stochastic_physics PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libstochastic_physics.a )
set_target_properties( fv3atm PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libfv3atm.a )
set_target_properties( ccppphys PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libccpp_physics.a )
set_target_properties( mom6 PROPERTIES IMPORTED_LOCATION ${DEPEND_LIB_ROOT}/${CMAKE_INSTALL_LIBDIR}/libmom6.a )



ecbuild_bundle( PROJECT femps GIT "https://github.com/jcsda-internal/femps.git" TAG 1.2.0 )
if(UFS_APP MATCHES "^(ATMAERO)$" OR UFS_APP MATCHES "^(S2S)$" OR UFS_APP MATCHES "^(ATM)$")
option(ENABLE_FV3_JEDI_DATA "Obtain fv3-jedi test data from fv3-jedi-data repository (vs tarball)" ON)
ecbuild_bundle( PROJECT fv3-jedi-data GIT "https://github.com/JCSDA-internal/fv3-jedi-data.git" BRANCH feature/ufs_dom UPDATE ) # updated from develop Dec 05 2023
ecbuild_bundle( PROJECT fv3-jedi GIT "https://github.com/jcsda-internal/fv3-jedi.git" BRANCH feature/ufs_dom UPDATE ) # updated from develop Dec 12 2023
if(UFS_APP MATCHES "^(S2S)$")
ecbuild_bundle( PROJECT soca GIT "https://github.com/jcsda-internal/soca.git" BRANCH feature/ufs_dom UPDATE ) # updated from develop Dec 12 2023
add_dependencies(soca ufs-weather-model)
endif()
ecbuild_bundle( PROJECT fv3-jedi-data GIT "https://github.com/JCSDA-internal/fv3-jedi-data.git" BRANCH develop )
ecbuild_bundle( PROJECT fv3-jedi GIT "https://github.com/jcsda-internal/fv3-jedi.git" BRANCH develop UPDATE )
elseif(UFS_APP MATCHES "^(NG-GODAS)$")
ecbuild_bundle( PROJECT soca GIT "https://github.com/jcsda-internal/soca.git" BRANCH feature/ufs_dom UPDATE ) # updated from develop Dec 12 2023
ecbuild_bundle( PROJECT soca GIT "https://github.com/jcsda-internal/soca.git" BRANCH feature/ufs_dom UPDATE )
add_dependencies(soca ufs-weather-model)
else()
message(FATAL_ERROR "ufs-bundle unknown UFS_APP ${UFS_APP}")
endif()


Comment thread
mark-a-potts marked this conversation as resolved.
Outdated
# Build Doxygen documentation
# ---------------------------
option(BUILD_UFSJEDI_BUNDLE_DOC "Build documentation" OFF)
Expand Down
Loading