Document singularity & apptainer

[kinow]

Singularity 3.6.x resulted in not enough loop devices error (fixed in a PR, but probably in a newer version).

Tried latest apptainer, crashed running mHM.

Installing singularity-ce 3.11 now.

[kinow]

Same as 3.6:

singularity exec mhm.sif /opt/conda/bin/mhm --nml "mhm_${EVAL_PERIOD_START}_${EVAL_PERIOD_END}.nml"
+ singularity exec mhm.sif /opt/conda/bin/mhm --nml mhm_1993_1995.nml
FATAL:   container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available

[kinow]

Possibly related to:

sylabs/singularity#65
sylabs/singularity#67

Tried enabling the option for loop devices in the singularity configuration, same error. Now rebuilding the image with --sandbox.

[kinow]

Ah, I see. That created a folder to be chroot'ed. Interesting, but that'd be inconvenient to transfer to an HPC or another remote platform 😞

[kinow]

When the shared loopback device setting is enabled, then the workflow fails with:

  in     0.402 seconds.
 Failed to open file: data/test_domain/output_b1/mHM_Fluxes_States.nc
 Permission denied    

[kinow]

Testing the building sandboxes instructions from this doc now: https://www.nas.nasa.gov/hecc/support/kb/best-practices-for-running-singularity-on-nas-systems_659.html

[kinow]

One task is still getting permission denied when trying to access output files, even with sandbox. Weird.

Running each start date separately works fine.

1991

1993

So the problem is really when running the same singularity container in parallel.

[kinow]

(Digressing, but interesting error)

+ docker build --no-cache=true --tag auto-mhm-test-domains/mhm:v5.12.0 .
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 335B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/mambaorg/micromamba:bullseye-slim
#3 ERROR: error getting credentials - err: docker-credential-secretservice resolves to executable in current directory (./docker-credential-secretservice), out: ``
------
 > [internal] load metadata for docker.io/mambaorg/micromamba:bullseye-slim:
------
Dockerfile:1
--------------------
   1 | >>> FROM mambaorg/micromamba:bullseye-slim
   2 |     
   3 |     RUN micromamba install --yes --name base --channel conda-forge \
--------------------
ERROR: failed to solve: error getting credentials - err: docker-credential-secretservice resolves to executable in current directory (./docker-credential-secretservice), out: ``

EDIT: solved

service docker stop
rm ~/.docker/config.json
service docker start

https://stackoverflow.com/questions/71770693/error-saving-credentials-error-storing-credentials-err-exit-status-1-out

[kinow]

For later: https://www.ssec.wisc.edu/~jimd/Singularity/Singularity_Pattern_Practice.pdf

[kinow]

Gave up and copied the data directory, one new folder for each simulation date 🤷‍♂️

NOTE 1: look if this could be an issue for running mHM in the HPC later on
NOTE 2: ask mHM devs to allow specifying the output folder via command-line