aws s3 ls on non-existent prefix returns error code 1

[alexvanacker]

Confirm by changing [ ] to [x] below:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Issue is about usage on:

• Service API : I want to do X using Y service, what should I do?
• CLI : passing arguments or cli configurations.
• Other/Not sure.

Platform/OS/Hardware/Device
What are you running the cli on? Ubuntu but not relevant

Describe the question
When reading the code for the command aws s3 ls (

aws-cli/awscli/customizations/s3/subcommands.py

Line 563 in b5e331e

def _check_no_objects(self):

), I see that the return code is 1 when we give a full path (bucket and key) and that it does not exist.

This is not correctly detailed in the documentation. I would like to open an issue and a PR to suggest a documentation improvement. What is the process for pure documentation issues?

Logs/output
Not relevant here.

[kdaily]

Hi @alexvanacker, thanks for the offer to contribute! Can you let me know where the issue is? The AWS CLI documentation lives in different places.

[alexvanacker]

Ah can you tell me where it is so I can contribute? The idea is that the documentation doesn't show that aws s3 ls returns 1 when the object doesn't exist.

[kdaily]

Do you mean if you do something like:

aws s3 ls help

Which also shows on this page: https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

Or, does it exist in the developer guide, like here:

https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3.html

[alexvanacker]

@kdaily I was indeed talking about https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

[kdaily]

The examples section for that page is here:

https://github.com/aws/aws-cli/blob/develop/awscli/examples/s3/ls.rst

Thanks!

[kdaily]

Can you describe which example or part of the documentation is incorrect? Thanks!

[alexvanacker]

Oh sorry, this one left my radar because of "business emergencies" ;)

The part that is incomplete (more than incorrect) is:

The following ls command lists objects and common prefixes under a specified bucket and prefix. However, there are no objects nor common prefixes under the specified bucket and prefix:

aws s3 ls s3://mybucket/noExistPrefix

Output:

None

When there are no objects, or the prefixes don't exist, the output is indeed None, however, the return code is not 0 but 1.
This should be documented since some scripts rely on checking existence (does the file exist? If so do we want to overwrite it or not, etc.). In bash scripts, since we get a 1, we can assume there was an error which is not the case here.

[alexvanacker]

I tried editing the file you mentionned with my suggestion, but when running

make html

in the doc folder, I do not see my changes applied. Not sure where to go from there?

[kdaily]

Hi @alexvanacker, you can open a PR against develop for us to review. Thanks!

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[perfectrecdan]

Hi, I would propose reopening this. It causes trouble especially when you are trying to run aws s3 ls within a Github action in order to see if an S3 object exists.

Is there a better way of doing this? My use case is gating execution of an s3 sync based on whether the target folder exists. I don't want to execute a sync that will overwrite things.

alexvanacker summarized the issue pretty well, I would maybe go as far as to say the output should not be an error though. What do folks think? Is it a legit use case to call s3 ls for basically a "stat" goal?

Thanks!

[petamas]

I'm running into the same issue (aws s3 ls s3://mybucket/noExistPrefix returning exit code 1) while trying to use vcpkg's experimental "binary cache in s3" feature. This causes vcpkg to print a confusing error message, see microsoft/vcpkg#31932 .

@kdaily , is there a way to reopen this issue and fix it somehow on the aws-cli end? I see multiple possible fixing options:

• If this is the intended behaviour of the aws s3 ls command, it should be documented at https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/ls.html as part of "Example 3: Listing all prefixes and objects in a specific bucket and prefix".
• If this is not intended, and you're fine with breaking backwards compatibility, then aws-cli should be modified to return exit code 0 in case of not finding anything with the given prefix.
• Regardless of whether it is intended or not, if you want to preserve the behaviour because there may be scripts depending on it, a new command line option could be introduced (eg. --no-fail-on-nonexistant-prefix) to make aws s3 ls succeed even if there are 0 hits.

Is it possible to implement any of the above options? If so, which of them fits most into aws-cli's philosophy / direction of evolution?

[cenit]

I'm running into the same issue (aws s3 ls s3://mybucket/noExistPrefix returning exit code 1) while trying to use vcpkg's experimental "binary cache in s3" feature. This causes vcpkg to print a confusing error message, see microsoft/vcpkg#31932 .

@kdaily , is there a way to reopen this issue and fix it somehow on the aws-cli end? I see multiple possible fixing options:

• If this is the intended behaviour of the aws s3 ls command, it should be documented at https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/ls.html as part of "Example 3: Listing all prefixes and objects in a specific bucket and prefix".
• If this is not intended, and you're fine with breaking backwards compatibility, then aws-cli should be modified to return exit code 0 in case of not finding anything with the given prefix.
• Regardless of whether it is intended or not, if you want to preserve the behaviour because there may be scripts depending on it, a new command line option could be introduced (eg. --no-fail-on-nonexistant-prefix) to make aws s3 ls succeed even if there are 0 hits.

Is it possible to implement any of the above options? If so, which of them fits most into aws-cli's philosophy / direction of evolution?

this affects also me ☝🏻
any indication on how to best contribute?

[kolbe]

To @perfectrecdan or others trying to use this command in a script, you may wish to try using aws s3api list-objects instead.

aws s3api list-objects --bucket "$bucket" --prefix "$prefix" --max-items 1

[petamas]

FYI @cenit and anyone else following this issue because of vcpkg reporting a warning/error due to aws exiting with exit code 1: my PR addressing this got accepted, and the fix has been released as part of vcpkg 2024.06.15. Updating vcpkg to that version will get rid of the annoying messages.

[cenit]

yes, i noticed your work immediately.
thank you very much!!