|
4 | 4 |
|
5 | 5 | To start, you need a project defined in the [standard Portable Encapsulated Project (PEP) format](http://pep.databio.org). Start by [creating a PEP](https://pep.databio.org/en/latest/simple_example/). |
6 | 6 |
|
7 | | -## 2. Connect the PEP to looper |
| 7 | +## 2. Specify the Sample Annotation |
8 | 8 |
|
9 | | -### 2.1 Specify `output_dir` |
10 | | - |
11 | | -Once you have a basic PEP, you can connect it to looper. Just provide the required looper-specific piece of information -- `output-dir`, a parent folder where you want looper to store your results. You do this by adding a `looper` section to your PEP. The `output_dir` key is expected in the top level of the `looper` section of the project configuration file. Here's an example: |
| 9 | +This information generally lives in a `project_config.yaml` file. |
12 | 10 |
|
| 11 | +Simplest example: |
13 | 12 | ```yaml |
14 | | -looper: |
15 | | - output_dir: "/path/to/output_dir" |
| 13 | +pep_version: 2.0.0 |
| 14 | +sample_table: sample_annotation.csv |
16 | 15 | ``` |
17 | 16 |
|
18 | | -### 2.2 Configure pipestat |
19 | | -
|
20 | | -*We recommend to read the [pipestat documentation](https://pipestat.databio.org) to learn more about the concepts described in this section* |
21 | | -
|
22 | | -Additionally, you may configure pipestat, the tool used to manage pipeline results. Pipestat provides lots of flexibility, so there are multiple configuration options that you can provide in `looper.pipestat.sample` or `looper.pipestat.project`, depending on the pipeline level you intend to run. |
23 | | - |
24 | | -Please note that all the configuration options listed below *do not* specify the values passed to pipestat *per se*, but rather `Project` or `Sample` attribute names that hold these values. This way the pipestat configuration can change with pipeline submitted for every `Sample` if the PEP `sample_modifiers` are used. |
25 | | - |
26 | | -- `results_file_attribute`: name of the `Sample` or `Project` attribute that indicates the path to the YAML results file that will be used to report results into. Default value: `pipestat_results_file`, so the path will be sourced from either `Sample.pipestat_results_file` or `Project.pipestat_results_file`. If the path provided this way is not absolute, looper will make it relative to `{looper.output_dir}`. |
27 | | -- `namespace_attribute`: name of the `Sample` or `Project` attribute that indicates the namespace to report into. Default values: `sample_name` for sample-level pipelines `name` for project-level pipelines , so the path will be sourced from either `Sample.sample_name` or `Project.name`. |
28 | | -- `config_attribute`: name of the `Sample` or `Project` attribute that indicates the path to the pipestat configuration file. It's not needed in case the intended pipestat backend is the YAML results file mentioned above. It's required if the intended pipestat backend is a PostgreSQL database, since this is the only way to provide the database login credentials. Default value: `pipestat_config`, so the path will be sourced from either `Sample.pipestat_config` or `Project.pipestat_config`. |
29 | | - |
30 | | -Non-configurable pipestat options: |
31 | | - |
32 | | -- `schema_path`: never specified here, since it's sourced from `{pipeline.output_schema}`, that is specified in the pipeline interface file |
33 | | -- `record_identifier`: is automatically set to `{pipeline.pipeline_name}`, that is specified in the pipeline interface file |
34 | | - |
| 17 | +A more complicated example taken from [PEPATAC](https://pepatac.databio.org/en/latest/): |
35 | 18 |
|
36 | 19 | ```yaml |
37 | | -name: "test123" |
38 | | -pipestat_results_file: "project_pipestat_results.yaml" |
39 | | -pipestat_config: "/path/to/project_pipestat_config.yaml" |
| 20 | +pep_version: 2.0.0 |
| 21 | +sample_table: tutorial.csv |
40 | 22 |
|
41 | 23 | sample_modifiers: |
42 | | - append: |
43 | | - pipestat_config: "/path/to/pipestat_config.yaml" |
44 | | - pipestat_results_file: "RESULTS_FILE_PLACEHOLDER" |
45 | 24 | derive: |
46 | | - attributes: ["pipestat_results_file"] |
| 25 | + attributes: [read1, read2] |
47 | 26 | sources: |
48 | | - RESULTS_FILE_PLACEHOLDER: "{sample_name}/pipestat_results.yaml" |
49 | | -
|
50 | | -looper: |
51 | | - output_dir: "/path/to/output_dir" |
52 | | - # pipestat configuration starts here |
53 | | - # the values below are defaults, so they are not needed, but configurable |
54 | | - pipestat: |
55 | | - sample: |
56 | | - results_file_attribute: "pipestat_results_file" |
57 | | - config_attribute: "pipestat_config" |
58 | | - namespace_attribute: "sample_name" |
59 | | - project: |
60 | | - results_file_attribute: "pipestat_results_file" |
61 | | - config_attribute: "pipestat_config" |
62 | | - namespace_attribute: "name" |
63 | | -``` |
64 | | -## 3. Link a pipeline to your project |
65 | | - |
66 | | -Next, you'll need to point the PEP to the *pipeline interface* file that describes the command you want looper to run. |
67 | | - |
68 | | -### Understanding pipeline interfaces |
69 | | - |
70 | | -Looper links projects to pipelines through a file called the *pipeline interface*. Any looper-compatible pipeline must provide a pipeline interface. To link the pipeline, you simply point each sample to the pipeline interfaces for any pipelines you want to run. |
71 | | - |
72 | | -Looper pipeline interfaces can describe two types of pipeline: sample-level pipelines or project-level pipelines. Briefly, a sample-level pipeline is executed with `looper run`, which runs individually on each sample. A project-level pipeline is executed with `looper runp`, which runs a single job *per pipeline* on an entire project. Typically, you'll first be interested in the sample-level pipelines. You can read in more detail in the [pipeline tiers documentation](pipeline-tiers.md). |
73 | | - |
74 | | -### Adding a sample-level pipeline interface |
75 | | - |
76 | | -Sample pipelines are linked by adding a sample attribute called `pipeline_interfaces`. There are 2 easy ways to do this: you can simply add a `pipeline_interfaces` column in the sample table, or you can use an *append* modifier, like this: |
77 | | - |
78 | | -```yaml |
79 | | -sample_modifiers: |
80 | | - append: |
81 | | - pipeline_interfaces: "/path/to/pipeline_interface.yaml" |
82 | | -``` |
83 | | - |
84 | | -The value for the `pipeline_interfaces` key should be the *absolute* path to the pipeline interface file. The paths may also contain environment variables. Once your PEP is linked to the pipeline, you just need to make sure your project provides any sample metadata required by the pipeline. |
85 | | - |
86 | | -### Adding a project-level pipeline interface |
87 | | - |
88 | | -Project pipelines are linked in the `looper` section of the project configuration file: |
89 | | - |
90 | | -``` |
91 | | -looper: |
92 | | - pipeline_interfaces: "/path/to/project_pipeline_interface.yaml" |
93 | | -``` |
94 | | - |
95 | | -### How to link to multiple pipelines |
96 | | - |
97 | | -Looper decouples projects and pipelines, so you can have many projects using one pipeline, or many pipelines running on the same project. If you want to run more than one pipeline on a sample, you can simply add more than one pipeline interface, like this: |
98 | | - |
99 | | -```yaml |
100 | | -sample_modifiers: |
101 | | - append: |
102 | | - pipeline_interfaces: ["/path/to/pipeline_interface.yaml", "/path/to/pipeline_interface2.yaml"] |
103 | | -``` |
104 | | - |
105 | | -Looper will submit jobs for both of these pipelines. |
106 | | - |
107 | | -If you have a project that contains samples of different types, then you can use an `imply` modifier in your PEP to select which pipelines you want to run on which samples, like this: |
108 | | - |
109 | | - |
110 | | -```yaml |
111 | | -sample_modifiers: |
| 27 | + # Obtain tutorial data from http://big.databio.org/pepatac/ then set |
| 28 | + # path to your local saved files |
| 29 | + R1: "${TUTORIAL}/tools/pepatac/examples/data/{sample_name}_r1.fastq.gz" |
| 30 | + R2: "${TUTORIAL}/tools/pepatac/examples/data/{sample_name}_r2.fastq.gz" |
112 | 31 | imply: |
113 | | - - if: |
114 | | - protocol: "RRBS" |
115 | | - then: |
116 | | - pipeline_interfaces: "/path/to/pipeline_interface.yaml" |
117 | | - - if: |
118 | | - protocol: "ATAC" |
119 | | - then: |
120 | | - pipeline_interfaces: "/path/to/pipeline_interface2.yaml" |
121 | | -``` |
122 | | - |
123 | | - |
124 | | -## 5. Customize looper |
125 | | - |
126 | | -That's all you need to get started linking your project to looper. But you can also customize things further. Under the `looper` section, you can provide a `cli` keyword to specify any command line (CLI) options from within the project config file. The subsections within this section direct the arguments to the respective `looper` subcommands. So, to specify, e.g. sample submission limit for a `looper run` command use: |
127 | | - |
128 | | -```yaml |
129 | | -looper: |
130 | | - output_dir: "/path/to/output_dir" |
131 | | - cli: |
132 | | - run: |
133 | | - limit: 2 |
134 | | -``` |
135 | | - |
136 | | -or, to pass this argument to any subcommand: |
137 | | - |
138 | | -```yaml |
139 | | -looper: |
140 | | - output_dir: "/path/to/output_dir" |
141 | | - all: |
142 | | - limit: 2 |
143 | | -``` |
144 | | - |
145 | | -Keys in the `cli.<subcommand>` section *must* match the long argument parser option strings, so `command-extra`, `limit`, `dry-run` and so on. For more CLI options refer to the subcommands [usage](usage.md). |
| 32 | + - if: |
| 33 | + organism: ["human", "Homo sapiens", "Human", "Homo_sapiens"] |
| 34 | + then: |
| 35 | + genome: hg38 |
| 36 | + prealignment_names: ["rCRSd"] |
| 37 | + deduplicator: samblaster # Default. [options: picard] |
| 38 | + trimmer: skewer # Default. [options: pyadapt, trimmomatic] |
| 39 | + peak_type: fixed # Default. [options: variable] |
| 40 | + extend: "250" # Default. For fixed-width peaks, extend this distance up- and down-stream. |
| 41 | + frip_ref_peaks: None # Default. Use an external reference set of peaks instead of the peaks called from this run |
| 42 | +``` |
0 commit comments