Running multiple instances of a singularity alphafold - fails /etc/ld.so.cache

[apelin20]

I am trying to run multiple predictions of the multimer model simultaneously using the same singularity .sif build and I am wondering if that's the reason I am running into:

ldconfig: Renaming of /etc/ld.so.cache~ to /etc/ld.so.cache failed: No such file or directory
/app/run_alphafold.sh: line 3: 13439 Illegal instruction     (core dumped) python /app/alphafold/run_alphafold.py "$@"

Perhaps these warnings are related:

mkdir: cannot create directory /tmp/lock-gpu1: File exists
mkdir: cannot create directory /tmp/lock-gpu0: File exists

I have checked the input and it doesn't seem like anything is wrong with the fasta file.

Singularity built from:
singularity build alphafold_v2.1.1.sif docker://uvarc/alphafold:2.1.1

Thanks,
Adrian

[RodenLuo]

I can confirm that I can run the docker version as described in README for multiple multimer predictions at the same time without any error. On Ubuntu 18.04.6 LTS with 4 GPU cards. (I sometimes set the --gpu_devices flag, while sometimes don't.)

[truatpasteurdotfr]

how do you run the singularity file?

[apelin20]

I run it on an SGE based cluster, my current version of the alphafold.wrapper script I use with qsub is here https://pastebin.com/QGNFCnTN

About a week ago alphafold v2.1.1 was compiled on the docker of a university (before that I couldn't figure out how to get it on singularity, never even heard of singularity before this):
mkdir /wynton/scratch/tmp; mkdir /wynton/scratch/cache; export SINGULARITY_TMPDIR=/wynton/scratch/tmp;
export SINGULARITY_CACHEDIR=/wynton/scratch/cache;
singularity build alphafold_v2.1.1.sif docker://uvarc/alphafold:2.1.1

qsub_cmd:
qsub -P gladstone -q gpu.q -pe smp 1 ~/qsub_wrapper/alphafold.wrapper "/wynton/group/krogan/programs/alphafold/input_fasta/VacV-N2L-TSPAN15.fasta" "/wynton/group/krogan/programs/alphafold/output_alphafold_VacV3" "multimer"

I am now experimenting with increasing GPU card from smp1 to smp2.

I am doing somethings wrong because many jobs fail, either memory issues, or something with the HT step, and the newest:
RuntimeError: Internal: Could not find the corresponding function

[RodenLuo]

You might want to take a look at these two links:

https://sbgrid.org/wiki/examples/alphafold2
https://wiki.hpcc.msu.edu/display/ITH/Alphafold

Our IT guy helped set up the alphafold in a python virtual env on a slurm based cluster. He told me he referred to these links.

[truatpasteurdotfr]

@apelin20

• if you don't use the same output directory for all the submission, that should be ok
• you might need to add a different /tmp directory for each run if your SGE setup is using /tmp/logk-gpuN for registreing locks
  (site specific matter): add a line with:

 -B `mktemp -d`:/tmp \

• Illegal instruction sounds familiar if your compute is of the younger than the generation of the cpu used to build the container.
  (not much you can do, except rebuilding the hhsuite software n the container).
  docker/Dockerfile line 30: replace with:

&& cmake -DHAVE_SSE4_1=1 -DCMAKE_INSTALL_PREFIX=/opt/hhsuite .. \

You can try my docker image from ghcr.io/truatpasteurdotfr/alphafold:main

[Augustin-Zidek]

Closing this issue as we currently provide support only for running via Docker. You can find more information about running under Singularity here: #10 and #24.