Skip to content

Commit 5e3a051

Browse files
TomKellyGeneticsKelly-ST-HRIggabernetTomKellyGeneticsapeltzer
authored
Add Universc (#1706)
* initialise template for new module: "universc" * update UniverSC module updates metadata, docker container source, licensing, and citations * update UniverSC module unit tests and documentation * update UniverSC module add inputs, outputs and example calls for UniverSC and Cell Ranger v3.0.2 calls versions for UniverSC and Cell Ranger * initialise template for new module: "universc" * update UniverSC module updates metadata, docker container source, licensing, and citations * update UniverSC module unit tests and documentation * update UniverSC module add inputs, outputs and example calls for UniverSC and Cell Ranger v3.0.2 calls versions for UniverSC and Cell Ranger * resolve formatting issues for UniverSC module * resolve linting errors for UniverSC module * fix test jobs to call UniverSC version without errors * correct configuration for UniverSC test jobs * correct linting errors for UniverSC module * correct docker build files for UniverSC * correct syntax errors in cellranger version call * prettier docs for UniverSC * add output to test data for UniverSC module * update UniverSC module to restructured repo nf-core/modules#2141 remove files for restructed UniverSC module (avoids duplicate tests) * define separate outputs for Cell Ranger and UniverSC tests * remove TODO statements and update UniverSC meta.yml defines input and output variables and triggers automated tests * resolve minor linting issues with UniverSC * update paths to UniverSC module in test config * update container versions and tags for UniverSC module * simplifiy container configurations for UniverSC * update configuration for UniverSC tests (build Cell Ranger transcriptome reference first) * test UniverSC module with Cell Ranger references * update tests for UniverSC for restructured repository * update reference inputs for UniverSC module * set up references for cellranger OS test * resolve permissions errors for starting UniverSC * update input arguments for UniverSC and Cell Ranger OS (tests passing) * correct versions and checksums for UniverSC tests * resolves linting issues for UniverSC * resolves linting issues for UniverSC * update expected test outputs for UniverSC tests * migrate UniverSC tests to calling open source Cell Ranger uses Cell Ranger 3.0.2 OS implementation (MIT License) tests passing locally * migrate changes to source code to updated UniverSC container * update unit tests for UniverSC to correct output (using new container) * update output criteria for UniverSC unit tests * change output directory for UniverSC unit tests * test adding podman to GitHub actions (will revert if reviewers object to it) * correct test errors for Cell Ranger OS tests (UniverSC module) * array format for test checks (UniverSC and Cell Ranger OS) * remove unncessary files from UniverSC module * remove podman from automated testing * remove mentions of nf-core/universc container * call executable script from PATH in UniverSC container * migrate UniverSC module Cell Ranger OS count own directory * reorganise UniverSC submodules * update process names in UniverSC module for consistency * update formatting for UniverSC tests * update unit tests for UniverSC submodules running Cell Ranger OS 3.0.2 * reorganise UniverSC submodules to fit naming conventions * remove stub from UniverSC for testing * add universc/mkfastq to unit tests * correct syntax in cellranger module files * update expected output for universc and universc/count tests now consistent with cellranger/count module * correct syntax for universc/count meta.yaml (pass linting) * add stub to universc and universc/count * update unit tests for universc module * update unit tests for universc module * update unit tests for universc module * update unit tests for universc module * update universc/mkfastq test to run stub * update universc/mkfastq expected outputs when running stub * remove trailing whitespace (linting error) * restructure UniverSC main module * update configuration to run each UniverSC test once only * correct UniverSC unit test configuration * update name of tools in universc tests configuration * update expected output for Cell Ranger and UniverSC tests * updates unit tests for Cell Ranger and UniverSC uses contain for web summary HTML as suggested by @apeltzer nf-core/modules#1706 * updates unit tests for Cell Ranger for web summary HTML * updates unit tests for Cell Ranger for web summary HTML * updates unit tests for Cell Ranger for web summary HTML to use description * correct path to Cell Ranger test output * update container options for run UniverSC with singularity runs without root priviledges in writeable container * migrate UniverSC container to mirrored image at nfcore/universc:1.2.4 adds documentation for image build configuration discussed in nf-core/modules#1706 * remove redundant submodules from UniverSC with functions already supported by Cell Ranger module nf-core/modules#1706 * migrate UniverSC references to generate by Cell Ranger submodules * update test configuration for universc/launch * update expected outputs for UniverSC to use Cell Ranger references * update expected outputs for UniverSC to use Cell Ranger references * remove UniverSC submodules for mkref and mkgtf (already implement in Cell Ranger module) discussed in nf-core/modules#1706 * move universc/launch submodule to universc module * remove tests for UniverSC submodules for mkref and mkgtf (already implement in Cell Ranger module) discussed in nf-core/modules#1706 * move tests for universc/launch submodule to universc module * migrate universc/launch submodule to universc module * update paths in unit tests from universc/launch to universc * update documentation for UniverSC module * update paths in test config from universc/launch to universc * restore cellranger module (remove changes from PR 1706) * restore cellranger module (remove changes from PR 1706) * restore cellranger module (remove changes from PR 1706) * update style of documentation to pass linting * add podman to settings and docs (passes local test) * test podman configuration * test podman configuration * restore changes to testing (removes podman discussed in nf-core/modules#2675) * restore changes to other modules (removes cellranger discussed in nf-core/modules#2646) * update podman settings in UniverSC docs * update podman parameters * update container version for universc to stable release 1.2.5 * remove conda tests for universc (not supported) * update container version for universc to latest release 1.2.5.1 (run tests on pushed version on personal account) * update container version for universc to use nfcore/universc:1.2.5.1 mirror * exit logic for universc module that doesn't support conda consistent with other modules exit logic for modules that dont support conda nf-core/modules#2657 * trigger GitHub Actions test for tomkellygenetics/universc:1.2.5.1 * add log files to universc output directory (confirm running subroutines as expected) * correct UniverSC test configuration addresses singularity test issue https://github.com/nf-core/modules/actions/runs/3955706571/jobs/6774566021 * update configuration for singularity in universc tests * test running universc with singularity --fakeroot requires shadow-uidmap::newuidmap installed * update configuration for singularity in universc tests * debug GH Actions configuration for singularity in universc tests * test running singularity with —fakeroot write permissions * test singularity— * revert changes to singularity tests disables singularity for universc (image too large) * update container settings for universc allows running rootless podman or singularity using --runtime crun or --writable-tmpfs apptainer/singularity#3220 * test universc with singularity --writable-tmpfs * revert changes to singularity tests (--writable-tmpfs not supported on GH Actions) * update container settings for universc to call nfcore/universc:1.2.5.1 (pending mirrored version available) * update version in UniverSC citation --------- Co-authored-by: Simon Thomas Kelly <[email protected]> Co-authored-by: Gisela Gabernet <[email protected]> Co-authored-by: TomKellyGenetics <tomkellygenetics@gmail> Co-authored-by: Alexander Peltzer <[email protected]>
1 parent 38ec6bf commit 5e3a051

File tree

5 files changed

+322
-0
lines changed

5 files changed

+322
-0
lines changed

universc/CITATION.cff

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
cff-version: 1.2.0
2+
message: "If you use this software, please cite it as below."
3+
authors:
4+
- given-names: "S. Thomas"
5+
family-names: "Kelly"
6+
7+
affiliation: "Center for Integrative Medical Sciences, RIKEN, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Japan"
8+
orcid: "https://orcid.org/0000-0003-3904-6690"
9+
- family-names: "Battenberg"
10+
given-names: "Kai"
11+
12+
affiliation: "Center for Sustainable Resource Science, RIKEN, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Japan"
13+
orcid: "http://orcid.org/0000-0001-7517-2657"
14+
version: 1.2.5.1
15+
doi: 10.1101/2021.01.19.427209
16+
date-released: 2021-02-14
17+
url: "https://github.com/minoda-lab/universc"
18+
preferred-citation:
19+
type: article
20+
authors:
21+
- given-names: "S. Thomas"
22+
family-names: "Kelly"
23+
24+
affiliation: "Center for Integrative Medical Sciences, RIKEN, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Japan"
25+
orcid: "https://orcid.org/0000-0003-3904-6690"
26+
- family-names: "Battenberg"
27+
given-names: "Kai"
28+
29+
affiliation: "Center for Sustainable Resource Science, RIKEN, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Japan"
30+
orcid: "https://orcid.org/http://orcid.org/0000-0001-7517-2657"
31+
- family-names: "Hetherington"
32+
given-names: "Nicola A."
33+
affiliation: "Center for Integrative Medical Sciences, RIKEN, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Japan"
34+
orcid: "http://orcid.org/0000-0001-8802-2906"
35+
- family-names: "Hayashi"
36+
given-names: "Makoto"
37+
affiliation: "Center for Sustainable Resource Science, RIKEN, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Japan"
38+
orcid: "http://orcid.org/0000-0001-6389-4265"
39+
- given-names: "Aki"
40+
family-names: "Minoda"
41+
42+
affiliation: Center for Integrative Medical Sciences, RIKEN, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Japan"
43+
orcid: "http://orcid.org/0000-0002-2927-5791"
44+
doi: "10.1101/2021.01.19.427209"
45+
title: "UniverSC: a flexible cross-platform single-cell data processing pipeline"
46+
year: "2021"
47+
journal: "bioRxiv"
48+
start: 2021.01.19.427209
49+
volume:
50+
issue:
51+
month: 1

universc/CITATION.md

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
### Citation <span id="Citation"><span>
2+
3+
A submission to a journal and biorXiv is in progress. Please cite these when
4+
they are available. Currently, the package can be cited
5+
as follows:
6+
7+
Kelly, S.T., Battenberg, Hetherington, N.A., K., Hayashi, K., and Minoda, A. (2021)
8+
UniverSC: a flexible cross-platform single-cell data processing pipeline.
9+
bioRxiv 2021.01.19.427209; doi: [https://doi.org/10.1101/2021.01.19.427209](https://doi.org/10.1101/2021.01.19.427209)
10+
package version 1.2.5.1. [https://github.com/minoda-lab/universc](https://github.com/minoda-lab/universc)
11+
12+
```
13+
@article {Kelly2021.01.19.427209,
14+
author = {Kelly, S. Thomas and Battenberg, Kai and Hetherington, Nicola A. and Hayashi, Makoto and Minoda, Aki},
15+
title = {{UniverSC}: a flexible cross-platform single-cell data processing pipeline},
16+
elocation-id = {2021.01.19.427209},
17+
year = {2021},
18+
doi = {10.1101/2021.01.19.427209},
19+
publisher = {Cold Spring Harbor Laboratory},
20+
abstract = {Single-cell RNA-sequencing analysis to quantify RNA molecules in individual cells has become popular owing to the large amount of information one can obtain from each experiment. We have developed UniverSC (https://github.com/minoda-lab/universc), a universal single-cell processing tool that supports any UMI-based platform. Our command-line tool enables consistent and comprehensive integration, comparison, and evaluation across data generated from a wide range of platforms.Competing Interest StatementThe authors have declared no competing interest.},
21+
eprint = {https://www.biorxiv.org/content/early/2021/01/19/2021.01.19.427209.full.pdf},
22+
journal = {{bioRxiv}},
23+
note = {package version 1.2.5.1},
24+
URL = {https://github.com/minoda-lab/universc},
25+
}
26+
27+
```
28+
29+
```
30+
@Manual{,
31+
title = {{UniverSC}: a flexible cross-platform single-cell data processing pipeline},
32+
author = {S. Thomas Kelly, Kai Battenberg, Nicola A. Hetherington, Makoto Hayashi, and Aki Minoda},
33+
year = {2021},
34+
note = {package version 1.2.5.1},
35+
url = {https://github.com/minoda-lab/universc},
36+
}
37+
```

universc/README.md

+116
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# UniverSC
2+
3+
## Single-cell processing across technologies
4+
5+
UniverSC is an open-source single-cell pipeline that runs across platforms on various technologies.
6+
7+
## Maintainers
8+
9+
Tom Kelly (RIKEN, IMS)
10+
11+
Kai Battenberg (RIKEN CSRS/IMS)
12+
13+
Contact: <first name>.<family name>[at]riken.jp
14+
15+
## Implementation
16+
17+
This container runs Cell Ranger v3.0.2 installed from source on MIT License on GitHub with
18+
modifications for compatibility with updated dependencies. All software is installed from
19+
open-source repositories and available for reuse.
20+
21+
It is _not_ subject to the 10X Genomics End User License Agreement (EULA).
22+
This version allows running Cell Ranger v3.0.2 on data generated from any experimental platform
23+
without restrictions. However, updating to newer versions on Cell Ranger subject to the
24+
10X EULA is not possible without the agreement of 10X Genomics.
25+
26+
To comply with licensing and respect 10X Genomics Trademarks, the 10X Genomics logo
27+
has been removed from HTML reports, the tool has been renamed, and proprietary
28+
closed-source tools to build Cloupe files are disabled.
29+
30+
It is still suffient to generate summary reports and count matrices compatible with
31+
single-cell analysis tools available for 10X Genomics and Cell Ranger output format
32+
in Python and R packages.
33+
34+
## Usage
35+
36+
### Generating References
37+
38+
The Cell Ranger modules can be used to generate reference indexes to run UniverSC.
39+
Note that UniverSC requires the Open Source version v3.0.2 of Cell Ranger included
40+
in the nf-core/universc Docker image. The same module parameters can be run provided
41+
that the container is changed in process configurations (modify nextflow.config).
42+
43+
```
44+
process {
45+
46+
...
47+
withName: CELLRANGER_MKGTF {
48+
container = "nfcore/universc:1.2.5.1"
49+
}
50+
withName: CELLRANGER_MKREF{
51+
container = "nfcore/universc:1.2.5.1"
52+
}
53+
...
54+
}
55+
```
56+
57+
This will generate a compatible index for UniverSC using the same version of the
58+
STAR aligner and a permissive software license without and EULA.
59+
60+
### Container settings
61+
62+
The cellranger install directory must have write permissions to run UniverSC.
63+
To run in docker or podman use the `--user root` option in container parameters
64+
and for singularity use the `--writeable` parameter.
65+
66+
These are set as default in universc/main.nf:
67+
68+
```
69+
container "nfcore/universc:1.2.5.1"
70+
if (workflow.containerEngine == 'docker'){
71+
containerOptions = "--privileged"
72+
}
73+
if (workflow.containerEngine == 'podman'){
74+
containerOptions = "--runtime /usr/bin/crun --userns=keep-id --user root --systemd=always"
75+
}
76+
if (workflow.containerEngine == 'singularity'){
77+
containerOptions = "--writable"
78+
}
79+
```
80+
81+
Select the container engine with `nextflow --profile "docker"` or set the environment variable
82+
as one of the following before running nextflow.
83+
84+
```
85+
export PROFILE="docker"
86+
export PROFILE="podman"
87+
export PROFILE="singularity"
88+
```
89+
90+
Note that due to dependencies installed in a docker image, it is not possible to use conda environments.
91+
92+
## Disclaimer
93+
94+
We are third party developers not affiliated with 10X Genomics or any other vendor of
95+
single-cell technologies. We are releasing this code on an open-source license which calls Cell Ranger
96+
as an external dependency.
97+
98+
## Licensing
99+
100+
This package is provided open-source on a GPL-3 license. This means that you are free to use and
101+
modify this code provided that they also contain this license.
102+
103+
## Updating the package
104+
105+
The tomkellygenetics/universc:<VERSION> container is automatically updated with tomkellygenetics/universc:latest.
106+
107+
A stable release is mirrored at nfcore/universc:1.2.5.1 and will be updated as needed.
108+
109+
To build an updated container use the Dockerfile provided here:
110+
111+
[https://github.com/minoda-lab/universc/blob/master/Dockerfile](https://github.com/minoda-lab/universc/blob/master/Dockerfile)
112+
113+
Note that this uses a custom base image which is built with an open-source implementation of
114+
Cell Ranger v3.0.2 on MIT License and relies of Python 2. The build file can be found here:
115+
116+
[https://github.com/TomKellyGenetics/cellranger_clean/blob/master/Dockerfile](https://github.com/TomKellyGenetics/cellranger_clean/blob/master/Dockerfile)

universc/main.nf

+76
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
process UNIVERSC {
2+
tag "$meta.id"
3+
label 'process_medium'
4+
5+
// Exit if running this module with -profile conda / -profile mamba
6+
if (workflow.profile.tokenize(',').intersect(['conda', 'mamba']).size() >= 1) {
7+
exit 1, "UNIVERSC module does not support Conda. Please use Docker / Singularity / Podman instead."
8+
}
9+
container "nfcore/universc:1.2.5.1"
10+
if (workflow.containerEngine == 'docker'){
11+
containerOptions = "--privileged"
12+
}
13+
if ( workflow.containerEngine == 'podman'){
14+
containerOptions = "--runtime crun --userns=keep-id --systemd=always"
15+
}
16+
if (workflow.containerEngine == 'singularity'){
17+
containerOptions = "-B /var/tmp --writable-tmpfs"
18+
params.singularity_autoMounts = true
19+
}
20+
21+
input:
22+
tuple val(meta), path(reads)
23+
path reference
24+
25+
26+
output:
27+
tuple val(meta), path("sample-${meta.id}/outs/*"), emit: outs
28+
path "versions.yml" , emit: versions
29+
30+
when:
31+
task.ext.when == null || task.ext.when
32+
33+
script:
34+
def args = task.ext.args ?: ''
35+
def sample_arg = meta.samples.unique().join(",")
36+
def reference_name = reference.name
37+
def input_reads = meta.single_end ? "--file $reads" : "-R1 ${reads[0]} -R2 ${reads[1]}"
38+
"""
39+
universc \\
40+
--id 'sample-${meta.id}' \\
41+
${input_reads} \\
42+
--technology '${meta.technology}' \\
43+
--chemistry '${meta.chemistry}' \\
44+
--reference ${reference_name} \\
45+
--description ${sample_arg} \\
46+
--jobmode "local" \\
47+
--localcores ${task.cpus} \\
48+
--localmem ${task.memory.toGiga()} \\
49+
--per-cell-data \\
50+
$args 1> _log 2> _err
51+
52+
# save log files
53+
echo !! > sample-${meta.id}/outs/_invocation
54+
cp _log sample-${meta.id}/outs/_log
55+
cp _err sample-${meta.id}/outs/_err
56+
57+
cat <<-END_VERSIONS > versions.yml
58+
"${task.process}":
59+
cellranger: \$(echo \$(cellranger count --version 2>&1 | head -n 2 | tail -n 1 | sed 's/^.* //g' | sed 's/(//g' | sed 's/)//g' ))
60+
universc: \$(echo \$(bash /universc/launch_universc.sh --version | grep version | grep universc | sed 's/^.* //g' ))
61+
END_VERSIONS
62+
"""
63+
64+
65+
stub:
66+
"""
67+
mkdir -p "sample-${meta.id}/outs/"
68+
touch sample-${meta.id}/outs/fake_file.txt
69+
70+
cat <<-END_VERSIONS > versions.yml
71+
"${task.process}":
72+
cellranger: \$(echo \$(cellranger count --version 2>&1 | head -n 2 | tail -n 1 | sed 's/^.* //g' | sed 's/(//g' | sed 's/)//g' ))
73+
universc: \$(echo \$(bash /universc/launch_universc.sh --version | grep version | grep universc | sed 's/^.* //g' ))
74+
END_VERSIONS
75+
"""
76+
}

universc/meta.yml

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: "universc"
2+
description: Module to run UniverSC an open-source pipeline to demultiplex and process single-cell RNA-Seq data
3+
keywords:
4+
- demultiplex
5+
- align
6+
- single-cell
7+
- scRNA-Seq
8+
- count
9+
- umi
10+
tools:
11+
- "universc":
12+
description: "UniverSC: a flexible cross-platform single-cell data processing pipeline"
13+
homepage: "https://hub.docker.com/r/tomkellygenetics/universc"
14+
documentation: "https://raw.githubusercontent.com/minoda-lab/universc/master/man/launch_universc.sh"
15+
tool_dev_url: "https://github.com/minoda-lab/universc"
16+
doi: "https://doi.org/10.1101/2021.01.19.427209"
17+
licence: ["GPL-3.0-or-later"]
18+
19+
input:
20+
- meta:
21+
type: map
22+
description: |
23+
Groovy Map containing sample information
24+
e.g. [ id:'test', single_end:false ]
25+
- reads:
26+
type: file
27+
description: FASTQ or FASTQ.GZ file, list of 2 files for paired-end data
28+
pattern: "*.{fastq,fq,fastq.gz,fq.gz}"
29+
30+
output:
31+
- outs:
32+
type: file
33+
description: Files containing the outputs of Cell Ranger
34+
pattern: "sample-${meta.id}/outs/*"
35+
- versions:
36+
type: file
37+
description: File containing software version
38+
pattern: "versions.yml"
39+
40+
authors:
41+
- "@kbattenb"
42+
- "@tomkellygenetics"

0 commit comments

Comments
 (0)