Duplicate S3 access log request IDs and glue deduping

[mikeplem]

We are testing version 6.0.0 of the tool using Glue 3.0 and have noticed that some access log data is being deduped when files are converted into hive/parquet format.

An example from our access logs are

9d306c3478cf2e54f72d7f972c2e7090d30324572b78f1901d30a5d89e33cbe8 BUCKET_NAME [02/May/2023:19:19:55 +0000] 52.12.241.113 IAM_ARN_HERE 1C25EZNCB2HBMQQY BATCH.DELETE.OBJECT f1683055189142x766494105173435800/IMG_1138.jpeg - 204 - - - - - - - - Pad2oayPBK9Yqw9/BWjhgn84fAsgRK7OTjjRFTy8Nuzlr27Ou+InFTsEf3eJsOaOkr2jw9xLBUa6d1tHwjf+xg== SigV2 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.amazonaws.com TLSv1.2 - -
9d306c3478cf2e54f72d7f972c2e7090d30324572b78f1901d30a5d89e33cbe8 BUCKET_NAME [02/May/2023:19:19:55 +0000] 52.12.241.113 IAM_ARN_HERE 1C25EZNCB2HBMQQY REST.POST.MULTI_OBJECT_DELETE - "POST /BUCKET_NAME/?delete HTTP/1.1" 200 - 305 - 29 - "-" "-" - Pad2oayPBK9Yqw9/BWjhgn84fAsgRK7OTjjRFTy8Nuzlr27Ou+InFTsEf3eJsOaOkr2jw9xLBUa6d1tHwjf+xg== SigV2 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.amazonaws.com TLSv1.2 - -

The Athena query output is as follows

#	bucket_owner	bucket	time	remote_ip	requester	request_id	operation	key	request_uri	http_status	error_code	bytes_sent	object_size	total_time	turnaround_time	referrer	user_agent	version_id	host_id	signature_version	cipher_suite	authentication_type	host_header	tls_version	year	month	day
1	9d306c3478cf2e54f72d7f972c2e7090d30324572b78f1901d30a5d89e33cbe8	BUCKET_NAME	2023-05-02 19:19:55.000	52.12.241.113	IAM_ARN_HERE	1C25EZNCB2HBMQQY	REST.POST.MULTI_OBJECT_DELETE		POST /BUCKET_NAME/?delete HTTP/1.1	200		305		29					Pad2oayPBK9Yqw9/BWjhgn84fAsgRK7OTjjRFTy8Nuzlr27Ou+InFTsEf3eJsOaOkr2jw9xLBUa6d1tHwjf+xg==	SigV2	ECDHE-RSA-AES128-GCM-SHA256			TLSv1.2 - -	2023	05	02

This is concerning because the Athena query output does not show which file was deleted. It appears the second s3 access log entry is overwriting the first when the file is converted.

• Is this expected?
• Is there a way to log both access log entries?

Thank you for taking the time to look into this.