The Great Library Shuffle

[bschroeter]

As per meeting with @JhanSrbinovsky and @ccarouge, we are working to reconcile the source code between the CABLE and UM7 repositories, with the purpose of generating a generic library object for use between applications.

Following my notes, I've started putting together a script template to handle the shuffle of code between repositories. This issue will serve as a discussion point.

#!/bin/bash

# Set configuration
WORK_DIR="~/work/library_shuffle"
CABLE_DIR="{WORK_DIR}/cable"
UM7_DIR="{WORK_DIR}/um7"
CABLE_REPO="[email protected]:CABLE-LSM/CABLE.git"
UM7_REPO="[email protected]:ACCESS-NRI/UM7.git"

mkdir -p $WORK_DIR

# Check everything out
cd $WORK_DIR
git clone $CABLE_REPO $CABLE_DIR
git clone $UM7_REPO $UM7_DIR

# Rename src/coupled/esm16 to src/coupled/esm
mv $CABLE_DIR/src/coupled/esm16 $CABLE_DIR/src/coupled/esm

# Replace the files in this directory with those from the UM7 repo
rsync -p $UM7_DIR/src/atmosphere/CABLE/src/coupled/ESM1.5/* $CABLE_DIR/src/coupled/esm/

# Create a new directory for the ESM1.5 files in the UM7 repo
UM7_ESM15_BACKUP=$UM7_DIR/src/atmosphere/land_surface/cable/ESM1.5
mkdir -p $UM7_ESM15_BACKUP

# Move the files from CABLE/ESM1.5 not in esm16 to the new directory
rsync -p $CABLE_DIR/src/coupled/ESM1.5/* $UM7_ESM15_BACKUP/

# NOTE: We need to manually remove those that are in esm16 at this point

# Remove ESM1.5 dir from CABLE
rm -r $CABLE_DIR/src/coupled/ESM1.5

@JhanSrbinovsky and @ccarouge , please make your adjustments as you see fit and I will finalise the script and handle the shuffle.

Cheers, Ben

[ccarouge]

I think it is going to be less easy than that.

The split of ESM1.5/ into 3 locations is going to be more tricky than that. 3 locations because we also need to create $CABLE_DIR/src/shared/

Here is what I have:

Put in $CABLE_DIR/src/shared/, delete from ESM1.5/, offline/ and/or esm16/. Use version in offline/
cable_LUC_EXPT.F90
cable_phenology.F90
casa_ncdf.F90
casa_offline_inout.F90

Need to add the shared/ directory to the CMakeLists.txt file for the library and offline compilations.

Files from $UM7_DIR/src/atmosphere/CABLE/src/coupled/ESM1.5 to put in $UM7_DIR/src/atmosphere/land_surface/cable/
casa_types.F90
pack_mod_cbl.F90
cable_rad_driver.F90
cable_um_init.F90
cable_um_init_subrs.F90
cable_um_tech.F90
allocate_soil_params_cbl.F90
allocate_veg_params_cbl.F90
cable_cbm.F90
cable_explicit_driver.F90
cable_hyd_driver.F90
cable_implicit_driver.F90
cbl_um_init_soil.F90
cbl_um_init_soilsnow.F90
cbl_um_update_soilsnow.F90
casa_um_inout.F90 (currently in esm16/ but it needs to be deleted from there)

Put in $CABLE_DIR/src/coupled/esm/, copied from $UM7_DIR/src/atmosphere/CABLE/src/coupled/ESM1.5/
cable_pft_params.F90
cable_soil_params.F90
cable_surface_types.F90
casa_landuse.F90
cable_define_types.F90
cable_iovars.F90

casa_landuse.F90 seems to be a CABLE science file but it is only stored under /src/coupled. Is it because we never run with the landuse scheme in offline? Should we keep it in $UM7_DIR instead then?

Note: after the first move, there might be some files moving to other locations. For example, I would think some of the files in the $UM7_DIR could end up in the $CABLE_DIR/src/coupled/shared if AM3 using the same version.

Second note: the library needs to compile files under CABLE/src/shared and CABLE/src/coupled/shared in addition to current set of files.

[bschroeter]

Thanks @ccarouge, I expected there would be a more complicated solution here. I think we should also include a readme in the various directories (src/shared in particular) to explain what is there and why.

I will revise the script and potentially set it up as a PR so that we can correct any issues inline before we go and execute the shuffle.

[JhanSrbinovsky]

Technically Im on leave but I just looked at Claire's revision. I agree with most of it but seeing cable_iovars.F90 again made me wonder what's actually in this file. SO I looked at it and I vaguely remember this from circa CMIP5 days but I thought this is all offline IO stuff, why would we need it!! So I grepped - it turns out that it is needed in cable_climate which is the other troublesome beast. Turns out that climate is only USEd from the offline drivers - so we shouldn't need it at all. That crosses 2 off the list - although I will verify first that it (ESM1.6) does indeed build/run without it.

[ccarouge]

@JhanSrbinovsky I like the idea of the library building all of science/ rather than cherry picking stuff. If we want to reorganise the science/ stuff later on to simplify, that's fine. But I wouldn't place this as a priority. The reorganisation we are looking at here allows us to limit file duplications which is good. Once we have the library stuff working, we can look on reducing unwanted dependencies.

Also, for cable_iovars.F90, I've put it in $CABLE_dir/src/coupled/esm/ because a diff between $UM_dir/*/src/coupled/ESM1.5 and $CABLE_dir/src/offline says there are differences. I have not looked to know if the differences are meaningful or not but for the moment, they are not exact copies of each other.

[JhanSrbinovsky]

Note: The files in UM7/...../CABLE/coupled/...... are the most recent. I just noticed a difference between the cable_pft_params file here VS the one in CABLE:main. I've got cable.nml reading from the top_level in the UM. I'll look at the PFT params next. Although I am technically on leave. Its likely they'll end up in CABLE/src/shared in a few days.

[JhanSrbinovsky]

getting the cable_pft_params to read through a namelist should be fairly straight forward AND using the same version of the file that is currently in offline/. i.e. something which could be moved to a shared/ directory - and then it occurred. to me why I hesitated in doing the same thing before. It isnt shared with AM3, CM2, ESM1.5. My guess is is that CM2, ESM1.5 will be dead soon anyway. ESM1.5 being absorbed as an earlier version of ESM/ anyway. AND we can in principle do this same thing for AM3 so it will indeed be shared. The story will possibly change again when JAC comes along but I cant see that happening this year.

[JhanSrbinovsky]

@bschroeter - as im sure you're aware - I was getting the veg/soil params to read from a uaelist rather than being hard-wired. In the process cable_veg/soil related TYPE decs/ allocate files were converged meaning they could be moved to src/shared. A recurring issue in reconciling them was the two different cable_define_types files. I have been reconciling these and they are now almost working. Unfortunately it is the offline version that is causing the most problems as it is referenced throughout the offline IO.