[vcpkg-tool] AWS CLI tool error when trying to download non-existing cache from AWS

[AenBleidd]

Previously I was successfully using AWS to store vcpkg binary cache.
Now I get next error when trying to download non-existing binary cache from AWS (that is a normal case of a cache miss when the port was updated and not yet built to save the binary cache):
error : aws failed with exit code: (1)
And while I'm able to finish the build, because this error was thrown, the overall build status is 'Failed'.
I'm not sure, if this is an issue with the updated AWS CLI client or with the updated vcpkg-tool.
If I find any additional details - I'll add them to this ticket.

[AenBleidd]

Small update: this is definitely related to the updated vcpkg-tool, and updated AWS tool can't be blamed.

[AenBleidd]

I may assume, this issue was added here microsoft/vcpkg-tool@8d2923f by @ras0219-msft

[JonLiu1993]

@AenBleidd, thanks for posting this issue, could you please provide steps to reproduce the issue?

[AenBleidd]

Hm, I thought it's quite obvious, but ok.

1. Create an AWS S3 bucket.
2. Configure vcpkg to use binary cache from it (e.g. on GitHub Actions for Windows: echo "VCPKG_BINARY_SOURCES=clear;x-aws,s3://AWS_S3_BUCKET_ADDRESS/,readwrite" | Out-File -FilePath $Env:GITHUB_ENV -Encoding utf-8 -Append
3. Ensure AWS S3 bucket is empty
4. Install any package (e.g. vcpkg install openssl)
5. Check log for the error line error : aws failed with exit code: (1)
6. Verify that package was correctly built and saved to AWS S3
7. Try to install this package again (for example by running the same GitHub Actions CI again)
8. Check that no error error : aws failed with exit code: (1) in the log because there was no cache miss and the package was successfully downloaded as binary cache from the AWS.

If you run this from the cmake build process - everything is fine, you will just receive this error in the log, but it will be ignored.
The problem is that if you run this from the msbuild - there will be a build error at the end.

Example of the log of msbuild with the cache miss and thus failed: https://github.com/BOINC/boinc/actions/runs/5232615295/jobs/9447451031
Example of the log after just rerun (because in case of the failed build, all updated packages were still built and successfully uploaded to the AWS S3, and thus on the next rerun there was no cache miss anymore): https://github.com/BOINC/boinc/actions/runs/5232615295

This is not a blocker (at least for me, since I can just rerun the CI manually), but it's still annoying, because cache miss is an expected situation, and there should be no failed build.
And no, I cant convert errors to warnings in the msbuild configuration, since then it will not catch a real vcpkg build issues.

I believe, @ras0219-msft already knows how to reproduce this, but still if you need me to test the fix when provided or required some additional information - please let me know.

[AenBleidd]

@JonLiu1993, please update the tags in this ticket since I provided requested information.

[JonLiu1993]

@vicroms, Could you please take a look?

[AenBleidd]

@JonLiu1993, @ras0219-msft, @vicroms, anything I could help you with to move forward with this issue?

[vicroms]

Hi @AenBleidd I'll investigate the issue.

[AenBleidd]

Any updates on this?

[petamas]

I've also been hit with this. It seems like aws s3 ls fails wit exit code 1 in case the prefix is nonexistent. (See aws-cli issue here: aws/aws-cli#5544)

I think it would be better to use aws s3api list-objects-v2 to query whether the file exists in the bucket because it exits with success regardless of whether the file is found or not, and its output can be checked to get the result of the existence check (it'll print a JSON in case the file exists, and nothing if it doesn't.)

So, instead of this:

aws s3 ls "s3://<bucket>/<path>/<hash>.zip"

vpckg would call this:

aws s3api list-objects-v2 --bucket "<bucket>" --prefix "<path>/<hash>.zip"

The result would be either:

{
    "Contents": [
        {
            "Key": "<path>/<hash>.zip",
            "LastModified": "<whatever>",
            "ETag": "\"<whatever>\"",
            "Size": <whatever>,
            "StorageClass": "STANDARD"
        }
    ]
}

or nothing in case the file does not exist.

@vicroms , would this be a reasonable change to vcpkg? I'd be happy to prepare a pull request if it makes sense to you.

(Note to self: the change would concern the stat() and download_file() methods here: https://github.com/microsoft/vcpkg-tool/blob/7a9a8f902c8ca682ba0cb6349ea346c4a6554162/src/vcpkg/binarycaching.cpp#L1119C9-L1119C9; I just need to figure out how the return a descriptive warning message in case there's a cache miss.)

[cenit]

any updates here?

[petamas]

@cenit

any updates here?

My PR (microsoft/vcpkg-tool#1404) fixing this just got merged, and will be part of the next vcpkg release.