singularity issue, growing containers and the like

[hjbockholt]

Is your feature request related to a problem? Please describe.
In an HPC environment, it's difficult to have containers that need to grow and/or gain permissions after they have been built.
Running sMRIPrep version 0.12.2+0.g93b0f3f7.dirty:
* BIDS dataset path: /data.
* Participant list: ['000300655084'].
* Run identifier: 20230927-233236_342145cb-b7a2-49ed-855e-3121f1b51499.

Spatial References: <none>.

Downloading https://templateflow.s3.amazonaws.com/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz
45%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 14.7M/32.4M [00:00<00:00, 68.0MB/s]
Process Process-2:
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/templateflow/api.py", line 239, in _s3_get
f.write(data)
OSError: [Errno 28] No space left on device

In my build environment, I neither have sudo permission nor can I can 'root' access once I'm in a singularity container.
The issue is that the default container does not download the templates for tensorflow.
Strictly, speaking containers should not add to themselves after they have been built.

Describe the solution you'd like
Consider just having the container already have the templates.

Describe alternatives you've considered
I added the following to the Dockerfile.

RUN mkdir /opt/templateflow && wget https://raw.githubusercontent.com/nipreps/fmriprep/master/scripts/fetch_templates.py && \
    python fetch_templates.py --tf-dir /opt/templateflow && rm fetch_templates.py

however, I can not figure out 'where' the templateflow templates should exist?
It looks like it might be default to $Home/.cache/templateflow, but when I manually download to that location on a writable container it still tries to download the templates again. Frustrating...

Additional context
singularity run --writable-tmpfs --containall --cleanenv --pid -B "/data/collaboration/FBIRN/containers/tmp:/tmp:rw,/data/collaboration/FBIRN/containers/license.txt:/opt/freesurfer/license.txt:ro,/data/collaboration/FBIRN/phase3:/data:rw,/data/collaboration/FBIRN/phase3/derivatives/dockerout:/out:rw" **output_container2.sif** --participant-label sub-000300655084 --nprocs 1 --omp-nthreads 2 --mem-gb 10 --fs-subjects-dir /data --fs-no-reconall -w /tmp --fs-license-file /opt/freesurfer/license.txt /data /out/out participant

[effigies]

Can you try using singularity exec ... docker://nipreps/fmriprep:23.1.4 smriprep? Or even just use the --anat-only flag for fMRIPrep? That's generally better supported than smriprep.

[effigies]

Did you already see https://www.nipreps.org/apps/singularity/#templateflow-and-singularity?