Merge pull request #40 from pepkit/dev

Update looper and pipestat changelogs
pepkit · Jan 16, 2025 · 211e16b · 211e16b
2 parents 22b4033 + 54fb83b
commit 211e16b
Show file tree

Hide file tree

Showing 6 changed files with 67 additions and 5 deletions.
diff --git a/docs/eido/writing-a-schema.md b/docs/eido/writing-a-schema.md
@@ -75,7 +75,7 @@ This document defines the required an optional sample attributes for this pipeli
 
 In the above example, we listed `read1` and `read2` attributes as *required*. This will enforce that these attributes must be defined on the samples, but for this example, this is not enough -- these also must *point to files that exist*. Checking for files is outside the scope of JSON Schema, which only validates JSON documents, so eido extends JSON Schema with the ability to specify which attributes should point to files.
 
-Eido provides two ways to do it: `files` and `required_files`. The basic `files` is simply used to specify which attributes point to files, which are not required to exist. This is useful for tools that want to calculate the total size of any provided inputs, for example. The `required_files` list specifies that the attributes point to files that *must exist*, otherwise the PEP doesn't validate. Here's an example of specifying an optional and required input attribute:
+Eido provides two ways to do it: `sizing` and `tangible`. The basic `sizing` is simply used to specify which attributes point to files, which are not required to exist. This is useful for tools that want to calculate the total size of any provided inputs, for example. The `tangible` list specifies that the attributes point to files that *must exist*, otherwise the PEP doesn't validate. Here's an example of specifying an optional and required input attribute:
 
 ```yaml
 description: A PEP for ATAC-seq samples for the PEPATAC pipeline.
@@ -109,7 +109,7 @@ properties:
         read2:
           type: string
           description: "Fastq file for read 2 (for paired-end experiments)"
-      required_files:
+      tangible:
         - read1
       files:
         - read1

diff --git a/docs/looper/changelog.md b/docs/looper/changelog.md
@@ -2,7 +2,7 @@
 
 This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.
 
-## [2.0.0a2] -- 2024-09-30
+## [2.0.0] -- 2025-01-16
 
 This release breaks backwards compatibility for Looper versions < 2.0.0
 
@@ -12,14 +12,17 @@ This release breaks backwards compatibility for Looper versions < 2.0.0
 - looper cli parameters now take priority as originally intended [#518](https://github.com/pepkit/looper/issues/518)
 - fix divvy inspect
 - remove printed dictionary at looper finish [#511](https://github.com/pepkit/looper/issues/511)
+- fix [#536](https://github.com/pepkit/looper/issues/536)
+- fix [#522](https://github.com/pepkit/looper/issues/522)
+- fix [#537](https://github.com/pepkit/looper/issues/537)
+- fix [#534](https://github.com/pepkit/looper/issues/534)
 
 ### Changed
 - `--looper-config` is now `--config`, `-c`. [#455](https://github.com/pepkit/looper/issues/455)
 - A pipeline interface now consolidates a `sample_interface` and a `project_interface` [#493](https://github.com/pepkit/looper/issues/493)
 - Updated documentation for Looper 2.0.0, removing previous versions [pepspec PR #34](https://github.com/pepkit/pepspec/pull/34)
 - remove position based argument for divvy config, must use --config or run as default config
 
-
 ### Added
 - `looper init` tutorial [#466](https://github.com/pepkit/looper/issues/466)
 - looper config allows for `pephub_path` in pipestat config section of `.looper.yaml` [#519](https://github.com/pepkit/looper/issues/519)

diff --git a/docs/looper/developer-tutorial/writing-a-pipeline-interface.md b/docs/looper/developer-tutorial/writing-a-pipeline-interface.md
@@ -207,12 +207,15 @@ properties:
         sample_name: 
           type: string
           description: "Name of the sample"
+          minLength: 1 # set a minimum length required for this required attribute
         file_path: 
           type: string
           description: "Path to the input file to count"
+          minLength: 1 # set a minimum length required for this required attribute
         area_type:
           type: string
           description: "Name of the components of the country"
+          minLength: 1 # set a minimum length required for this required attribute
       required:
         - sample_name
         - file_path
@@ -223,7 +226,8 @@ required:
 
 This file specifies what inputs the pipeline uses, and what type they are.
 It is a [JSON Schema](https://json-schema.org/), which allows us to use this file to validate the inputs, which we'll cover later.
-For now, it defines that our input samples have 3 properties: `sample_name`, `file_path`, and `area_type`.
+For now, it defines that our input samples have 3 properties: `sample_name`, `file_path`, and `area_type`. 
+We also specify a minimum length required for these inputs with `minLength: 1`.
 
 ### Adapt the pipeline interface to use the input schema
 

diff --git a/docs/looper/how-to/rerun.md b/docs/looper/how-to/rerun.md
@@ -0,0 +1,47 @@
+# How to rerun samples
+
+## Introduction
+
+It is recommended that you have read and completed the basic [pipestat tutorial](../user-tutorial/user-pipestat.md) as well as [configuring pipestat backends](configure-pipestat-backends.md). 
+
+This document will briefly cover details about how to configure pipestat in looper:
+
+!!! success "Learning objectives"
+    - How do I rerun samples?
+
+There may be a time when you need to re-submit samples. Looper's `rerun` command can be used to achieve this. By default, the `rerun` command will gather any samples with a `failed` status and submit again.
+
+However, Looper does not create flags for samples. It only reads them. Creating flags is the job of the pipeline via either the pipeline manager (e.g. [pypiper](../../pypiper/README.md)) or pipestat.
+
+Therefore, to create submission flags, you must:
+
+1. Configure looper to use pipestat
+2. Add the `flag_file_dir` key and a flag directory path under the pipestat key within the looper config:
+```yaml title=".looper.yaml" hl_lines="6"
+pep_config: project_config_pipestat.yaml
+output_dir: results
+pipeline_interfaces:
+  - ./pipeline_interface1_sample_pipestat.yaml
+pipestat:
+  flag_file_dir: output/results_pipeline
+  results_file_path: tmp_pipestat_results.yaml
+```
+3. Ensure your pipeline is using pipestat to set the [status](../../pipestat/code/python-tutorial.md#pipeline-status-management) upon sample processing.
+
+```python
+currentPipestatManager.set_status(record_identifier="sample1", status_identifier="running")
+currentPipestatManager.set_status(record_identifier="sample2", status_identifier="failed")
+```
+More info on setting status can be found [here](../developer-tutorial/developer-pipestat.md#setting-and-checking-status).
+
+
+Now that your pipeline is setting statuses, you can resubmit failed jobs with:
+```shell
+looper rerun
+```
+
+By default, this command will _only_ submit failed jobs. However, you can use the `--ignore-flags` flag to resubmit all jobs.
+```shell
+looper rerun --ignore-flags
+```
+
diff --git a/docs/pipestat/changelog.md b/docs/pipestat/changelog.md
@@ -2,6 +2,13 @@
 
 This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.
 
+## [0.12.0] - 2025-01-16
+### Fixed
+- portable report now has proper file extension in messaging.
+- add exception to pipestat summarize if there are no results to report [#210](https://github.com/pepkit/pipestat/issues/210)
+- fix spaces in html files [#211](https://github.com/pepkit/pipestat/issues/211)
+- add output_dir parameter to psm.table
+
 ## [0.11.0] - 2024-10-02
 ### Fixed
 - for output schema, make samples an array type and nest under items [#204](https://github.com/pepkit/pipestat/issues/204)

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -153,6 +153,7 @@ nav:
         - Running project-level pipelines: looper/how-to/project-level-pipelines.md
         - Link similar results: looper/how-to/link-objects.md
         - Report objects:  looper/how-to/report-objects.md
+        - Rerun samples: looper/how-to/rerun.md
     - Pipeline Developer Docs:
       - Developer tutorial:
         - Writing a pipeline interface: looper/developer-tutorial/writing-a-pipeline-interface.md