Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add example of using GNU parallel to run tasks in parallel #1915

Merged
merged 5 commits into from
May 15, 2024

Conversation

amarao
Copy link
Contributor

@amarao amarao commented Feb 20, 2024

I found amazing synergy between GNU Parallel and just (basically, use parallel in --shebang mode), adding example into examples/.

@casey
Copy link
Owner

casey commented Feb 20, 2024

This is great! I think we should actually add this to the readme, since it seems super useful.

A few things:

  • There's a num_cpus() function, which can be used to set the number of jobs appropriately.
  • I think it's a good idea to use long flags instead of short flags, so --jobs instead of -j.
  • With --ungroup, parallel prints job output right away, instead of waiting for the job to finish, which lets you see when jobs start up.
  • /usr/bin/env will run parellel regardless of where it is.
  • I think we should call the recipe parallel, since churn is a bit uninformative.

With these changes, the recipe looks like this:

parallel:
  #!/usr/bin/env parallel --shebang --ungroup --jobs {{ num_cpus() }}
  echo task 1 start; sleep 1; echo task 1 done
  echo task 2 start; sleep 2; echo task 2 done
  echo task 3 start; sleep 3; echo task 3 done
  echo task 4 start; sleep 4; echo task 4 done

@amarao
Copy link
Contributor Author

amarao commented Feb 21, 2024

I've updated the example.

@amarao
Copy link
Contributor Author

amarao commented Feb 21, 2024

Oh, and I found how to force env and parallel to work together in shebang mode.

Thanks for feedback.

@casey casey enabled auto-merge (squash) May 15, 2024 01:11
@casey casey disabled auto-merge May 15, 2024 01:13
@casey casey enabled auto-merge (squash) May 15, 2024 01:14
@casey
Copy link
Owner

casey commented May 15, 2024

I moved this into the readme, since I think it'll be more discoverable there. Thanks for the PR!

@casey casey merged commit 1b62918 into casey:master May 15, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants