Support for AWS Batch Consumable Resources

[exai-sukh]

New Feature

AWS has introduced a new feature for AWS Batch, called "Consumable Resources". It would be great if Nextflow's AWS executor could support this. You may see additional details in the AWS blog post How to use rate-limited resources in AWS Batch jobs with resource-aware scheduling.

Use Case

I have workflows which need staggered submissions to reduce the likelihood of throttling errors. This feature would resolve such issues.

Suggested Implementation

Specify ConsumableResourceProperties within the process directive. This could be added to the SubmitJobRequest in

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy

Line 713 in 7d3cb2b

protected SubmitJobRequest newSubmitRequest(TaskRun task) {

[bentsherman]

If it's a new feature then it's likely only available in AWS SDK v2, which we haven't upgraded to yet. We plan to do that this year.

In the meantime, we do have some retry-with-backoff logic to handle rate limit errors. Is that not working for you?

[gbillings-profound]

Would love to see this implemented! We have a pipeline using commercially-licensed software in AWS Batch, and our license is for a set number of concurrent processes; implementing consumable resources seems like the cleanest way to manage the license seats efficiently. Other possible strategies are using process.maxForks or just capping the number of vCPUs for the compute environment, but they both have significant limitations.

[bentsherman]

Looking at the API docs, I guess we could have a process directive like this:

process dragen {
  consumableResources 'my-dragen-license': 1, 'other-license': 2

  // ...
}

Nextflow uses AWS SDK v2 now so it should be feasible

[gbillings-profound]

Something like that would be ideal for us!