Prebuilt Singularity Images

[aurelieGabriel]

Hi,
First, thank you for providing such a comprehensive pipeline for GRN analysis.
I attempted to build the various Singularity images to test the pipeline. However, I encountered errors while building some of the images, e.g. the scenicplus.sif and gretabench.sif.
Here is for example the error that I have when building the scenicplus image:

singularity build scenicplus.sif scenicplus.def
ERROR: Could not build wheels for datrie, which is required to install pyproject.toml-based projects
FATAL:   While performing build: while running engine: exit status 1

Would it be possible for you to share the prebuilt .sif files, for example, via Zenodo or another repository?
Thank your in advance for your help.
Best wishes,
Aurélie

[PauBadiaM]

Hi @aurelieGabriel,

Indeed! Give me a minute and I'll upload them 😉

[PauBadiaM]

With the latest commit (f4f0c00) now the snakemake pipeline should automatically download the images from zenodo so there is no need to compile them locally anymore. Let me know if you find any more issues!

[aurelieGabriel]

Hi @PauBadiaM,

Thank you very much for your quick reply and for adding the automatic download of the singularity images.

I could start testing the pipeline on the pbmc10x dataset. However, note that the .sif files should also be added as input of the rules in the dts folder. Otherwise, the .sif files are not ready for these rules to run.
The same error is encountered for some rules in gen.smk, e.g. gen_ann_pando, in the p2g_granie, and in the gen_genome_celloracle rule.

Also, in my hands, running the pipeline with the current slurm config files does not work with latest versions of Snakemake, since Snakemake8.0 and higher versions are quite different from previous snakemake versions.
With Snakemake version 7.29.0 and using the slurm config files from your repositories, I obtained the following error:

snakemake --profile config/slurm/ dts/pbmc10k/cases/all/mdata.h5mu
snakemake: error: Couldn't parse config file: 'tuple' object has no attribute 'safe_load'

However using a custom config file and the following command line the pipeline could start:

snakemake -j 30 -kps workflow/Snakefile --configfile config/config.yaml --use-singularity --cluster-config config/cluster.yml  --cluster "sbatch -A {cluster.account} -p {cluster.partition} -N {cluster.N} -t {cluster.time}  --job-name {cluster.name} --mem {cluster.mem} --cpus-per-task {cluster.cpus-per-task} --output {cluster.output} --error {cluster.error}" dts/pbmc10k/cases/all/mdata.h5mu

Here is the content of the cluster.yml file that I used:

  description: |
    Cluster config for greta pipeline

  __default__:
    cpus-per-task : "1"
    N : "1"
    time : "1:00:00"
    name : "{rule}"
    partition : "cpu"
    account : "XX"
    mem : "64000"
    output : "cluster/snakemake/{rule}.output"
    error : "cluster/snakemake/{rule}.error"

I could obtain the following files:

dts/pbmc10k/cases/all/mdata.h5mu
dts/pbmc10k/cases/all/runs/pando.pre.h5mu
dts/pbmc10k/cases/all/runs/pando.granie.p2g.csv

However I could not compute CellOracles's tfb step on GRaNIE's p2g due to this error in the tfb_celloracle rule:

FileNotFoundError: could not find hg38.fa(.gz) in genome_dir ".../dbs/hg38/gen/genome/celloracle/hg38"

This last error is probably not related to this github issue, I can open a new issue related to this if needed.
Thank you in advance for your help.

Best wishes,
Aurélie