Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detecting errors in data unstaging #5345

Merged
merged 13 commits into from
Dec 2, 2024
Merged

Detecting errors in data unstaging #5345

merged 13 commits into from
Dec 2, 2024

Conversation

jorgee
Copy link
Contributor

@jorgee jorgee commented Sep 30, 2024

close #3711

  • Enables fast failure (set -e) at unstage part and removes "|| true" statement in the unstage.
  • Splits unstage in control and task outputs.
  • Unstage of control outputs after task outputs to see errors in .command.err/out
  • Command to skip task outputs unstaging is 'true' instead of null (Google Life sciences part)

Copy link

netlify bot commented Sep 30, 2024

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 96a851d
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/674de3a05810070008c21608

@jorgee jorgee changed the title Test 3711 Detecting errors in data unstaging Sep 30, 2024
@bentsherman bentsherman requested a review from pditommaso October 2, 2024 15:25
@jorgee jorgee marked this pull request as draft October 9, 2024 08:55
@jorgee jorgee marked this pull request as ready for review October 9, 2024 11:08
@nextflow-io nextflow-io deleted a comment from bentsherman Nov 20, 2024
Signed-off-by: Paolo Di Tommaso <[email protected]>
Comment on lines +102 to +108
## Capture possible errors.
## Can be caused either by the task script, unstage script or after script if defined
local last_err=$?
## capture the task error first or fallback to unstage error
local exit_status=${nxf_main_ret:=0}
[[ ${exit_status} -eq 0 && ${nxf_unstage_ret:=0} -ne 0 ]] && exit_status=${nxf_unstage_ret:=0}
[[ ${exit_status} -eq 0 && ${last_err} -ne 0 ]] && exit_status=${last_err}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still very hard to decode, it would be nice to be more verbose about the different of last_err vs nxf_main_ret vs nxf_unstage_ret and the general logic.

Copy link
Member

@pditommaso pditommaso Dec 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok reported as separate issue

@pditommaso pditommaso merged commit 3c8e602 into master Dec 2, 2024
36 checks passed
@pditommaso pditommaso deleted the test_3711 branch December 2, 2024 17:42
christopher-hakkaart pushed a commit to christopher-hakkaart/nextflow that referenced this pull request Dec 3, 2024
Signed-off-by: jorgee <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Co-authored-by: Paolo Di Tommaso <[email protected]>
Signed-off-by: Christopher Hakkaart <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Validation of task outputs
2 participants