Recursively downloading S3 directory restored from Glacier doesn't work

[Mridul-Chaudhary]

New feature

For rnasplice pipeline, I wanted to provide my previous salmon output as an input as suggested here

So I prepared input sheet like this:

sample,condition,salmon_results
xyz,case,s3://bucket/path/salmon/xyz

where xyz directory is in Glacier storage class but has been restored

xyz/
├── aux_info
│   ├── ambig_info.tsv
│   ├── expected_bias.gz
│   ├── fld.gz
│   ├── meta_info.json
│   ├── observed_bias_3p.gz
│   └── observed_bias.gz
├── cmd_info.json
├── lib_format_counts.json
├── libParams
│   └── flenDist.txt
├── logs
│   └── salmon_quant.log
├── quant.genes.sf
└── quant.sf

Each of the file inside xyz has been individually restored and has been tested for download. It does not require --force-glacier-transfer option.

But my sample-sheet above forces nxf_s3_download() function to attempt recursive download on xyz directory which is not downloadable without specifying --force-glacier-transfer option

Example:
aws s3 cp s3://bucket/path/salmon/xyz ./xyz/ --recursive does not work
Error:
warning: Skipping file s3://bucket/path/salmon/xyz/aux_info/ambig_info.tsv. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 download help for additional parameter options to ignore or force these transfers

aws s3 cp s3://bucket/path/salmon/xyz ./xyz/ --recursive --force-glacier-transfer works

Usage scenario

Provide restored glacier directory paths as input

Suggest implementation

Similar to a change made here , --quiet was changed to --only-show-errors , --force-glacier-transfer option could be made a default in case the s3 object is not in standard storage

Or,
Introduce an optional setting similar to aws.client.glacierAutoRetrieval, say aws.client.glacierForceTransfer, wihch when true will add --force-glacier-transfer to nxf_s3_download() function

[bentsherman]

Hi @ludirm93 , we do not plan to support automatic Glacier restore and we have removed the glacierAutoRetrieval option. See #4511 (comment) for the discussion on this point.

Instead, I recommend that you write a custom process to restore any Glacier objects all at once, to minimize the amount of time waiting for the Glacier restore

[Mridul-Chaudhary]

Hi @bentsherman
Thank you for your response!
May be I was not able to convey my request..
The s3 files have already been restored by me manually..

nextflow is not able to download even the pre-restored because in nxf_s3_download() function, it would require an aws cli option "--force-glacier-transfer"

Worst case, I would need to make a duplicate copy of my data in S3 and keep it in standard storage

[bentsherman]

That is strange, it should be able to download the S3 object like normal. How exactly did you restore it from Glacier?

[Mridul-Chaudhary]

For the salmon output folder tree shown above, I individually executed restore-object command for each file.

aws s3api restore-object --bucket bucket --key path/salmon/xyz/aux_info/ambig_info.tsv --restore-request '{"Days":17,"GlacierJobParameters":{"Tier":"Standard"}}'

When I try to download the files outside of nextflow, to test if it is downloadable,
I can download each of the files with by using complete key

aws s3 cp s3://bucket/path/salmon/xyz/quant.sf ./

I need to provide entire salmon output directory xyz as an input.

aws s3 cp s3://bucket/path/salmon/xyz/ ./ --recursive does not work.

aws proposes to add "--force-glacier-transfer" option in recursive download of a prefix

The nxt_s3_download() function used to get these restored files is mentioned below:

nxf_s3_download() {
local source=$1
local target=$2
local file_name=$(basename $1)
local is_dir=$(/home/ec2-user/miniconda/bin/aws --region eu-west-1 s3 ls $source | grep -F "PRE ${file_name}/" -c)
if [[ $is_dir == 1 ]]; then
/home/ec2-user/miniconda/bin/aws --region eu-west-1 s3 cp --only-show-errors --recursive "$source" "$target"
else
/home/ec2-user/miniconda/bin/aws --region eu-west-1 s3 cp --only-show-errors "$source" "$target"
fi
}

It recognizes my input path to be a directory and thus goes with recursive download option which in this case won't work because it needs --force-glacier-transfer

I do not know if there is a way to pass this to nxf_s3_download()

[Mridul-Chaudhary]

Additionally,
Listing recursively on a restored directory works without --force-glacier-transfer.

aws s3 ls s3://bucket/path/salmon/xyz/ --recursive

So another solution without passing --force-glacier-transfer could be,

if [[ $is_dir == 1 ]]; then
list all files
download each of the above

[bentsherman]

So this problem only happens when recursively downloading a directory, not with files

[Mridul-Chaudhary]

Thanks for re-opening the issue.
Yes, that is correct..
This would affect pipelines or modules that need an entire directory as input.
For me, it was RNAsplice pipeline with salmon results as input.

[bentsherman]

Seems like if the aws cli can recursively list the directory but not download it, it is a bug with the aws cli.

In the meantime, you can work around this issue by specifying the files in the directory as inputs rather than the directory itself. It's slightly more verbose but much better in the long run.

[MrOlm]

@bentsherman - it's not quite a bug (it's how they intend it to be- https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html), but I agree it's a really strange implementation choice.

Just wanted to say I'm hitting the exact same issue as @Mridul-Chaudhary , but it's not quite clear to be how to switch from using directories to listing all files in a directory and using that instead.

Thanks,
Matt

[bentsherman]

There is a way, but it ain't pretty if the input directory is deeply nested. Instead of providing the directory as a single input, you can enumerate all of the files in the directory so that they are downloaded individually instead of a single recursive download

I can understand the AWS CLI not wanting to recursively restore Glacier objects, but I don't understand why they are restricting objects that are already restored. Those objects should behave like regular objects

[MrOlm]

Totally agree with you on the AWS CLI implementation, and I was as surprised as you are to find they implemented it this way, but it is what it is.

Do you have any example code you could maybe point me to around this workaround? I'm just not sure how it would work 1) generating a list of these files in nextflow clode and 2) passing this list of files of variable length and order to a nextflow process.

Thanks!
Matt