Slurm-scripts

[edyoshikun]

This PR adds examples on how to use slurm for parallel reconstructions for birefringence and phase.

Both pipilines convert the data to zarr, create an empty zarr store for the output and parallelize the reconstructions per position to simultaneously write to the output zarr.

These scripts run using our zenodo dataset, but bigger datasets really take an advantage using HPC.

[codecov]

Codecov Report

Merging #337 (20347ff) into main (add3280) will decrease coverage by 0.32%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##            main    #337      +/-   ##
========================================
- Coverage   2.06%   1.75%   -0.32%     
========================================
  Files         21      20       -1     
  Lines       4888    4841      -47     
========================================
- Hits         101      85      -16     
+ Misses      4787    4756      -31     

see 5 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[deprecated-napari-hub-preview-bot]

Preview page for your plugin is ready here:
https://preview.napari-hub.org/mehta-lab/recOrder/337
Updated: 2023-04-17T17:38:48.602463

[edyoshikun]

I think I need to add documentation on how to run the code.
The scripts are bash scripts and the phase or birefringence reconstructions can be executed by bash batch_bire_recon.sh or bash batch_phase_recon.sh respectively. Thank you both!

[mattersoflight]

Thanks for these utilities @edyoshikun. Yes, a quick how-to on staging the data will be useful. These can be comments in specific scripts or a markdown guide linked in the docs.

Have you already implemented a utility that allows the reconstruction of data parallelized by positions, preceded by or followed by, the creation of metadata at all positions within the zarr store? Is iohub's metadata API now modular enough to do so?

[ziw-liu]

Have you already implemented a utility that allows the reconstruction of data parallelized by positions, preceded by or followed by, the creation of metadata at all positions within the zarr store? Is iohub's metadata API now modular enough to do so?

I think this file does this (although using iohub internals which are now wrapped in public API, see following comment).

[talonchandler]

@edyoshikun this PR no longer needs a review, correct? Can you change to draft or close?

[talonchandler]

Deprecated by recent work on mantis