aws s3 sync with include/exclude does wrongly report objects do not exist and reupload them

[mapk-amazon]

Describe the bug

The s3 sync command is wrongly identifying a file as not yet uploaded when include/excludes are provided.

Expected Behavior

I expect sync to behave the same in those two situation:

Syncing a directory with a single file vs. _Syncing a directory containing a single file with include/exclude conditions satisfied by the file

Current Behavior

The current behaviour does not recognize already uploaded files and is overwriting them, when including include/exclude commands.

Reproduction Steps

I have the following directory structure

.
├── testfile
│   └── ec2-user.txt

And I try to upload the file with sync which I should recognize that the object is already on S3 and not reupload it again. It works when i do

aws s3 sync  testfile/  s3://$bucketname/ 

Starting the same command does not try to upload anything after the first run (or when the file has changed).

But once I specify exclude and include (with $abspathfile being the absolute path of the single file)

aws s3 sync  testfile/  s3://$bucketname/  --exclude "*" --include "$abspathfile" 

it always uploads the file. When running it with debug, it does listV2 calls and reports (wrongly)

2024-09-19 18:11:49,138 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: XXXXX/testfile/ec2-user.txt -> $bucketname/ec2-user.txt, file does not exist at destination

Possible Solution

AWS CLI should recognize already uploaded files

Additional Information/Context

No response

CLI version used

aws-cli/2.17.54 Python/3.12.6 Linux/6.1.79-99.167.amzn2023.x86_64 exe/x86_64.amzn.2023

Environment details (OS name and version, etc.)

Amazon Linux 2023

[mapk-amazon]

I believe the issue is here

aws-cli/awscli/customizations/s3/filters.py

Line 106 in aeb3ba5

def call(self, file_infos):

It applies the filter for the local filesystem to the S3 bucket. It excludes my file because it fits the * pattern, but does not "reinclude" it because the path on S3 is different than my local path. This leads the CLI to believe that the file does not exist on my S3 bucket and thus will reupload the file.

[RyanFitzSimmonsAK]

Hi, thanks for reporting this issue. I was able to reproduce the behavior, and am working on root causing and fixing it. I'll put any updates in this issue. Thanks!

[RyanFitzSimmonsAK]

I don't think this is an issue with the filters; those appear to be consistent between using absolute path and relative path. Additionally, the logic for determining if a file exists in an S3 bucket is separate from the filters. The ListObjectsV2 response is also correct, so it's possibly an issue with the CLI interpreting that response. Regardless, thank you for reporting this bug.