[destination-redshift] does not name staging S3 files correctly leading to missing data #65954
Description
Connector Name
destination-redshift
Connector Version
3.5.3
What step the error happened?
During the sync
Relevant information
According to the documentation, S3 Filename pattern supports {timestamp:micros} however when we used that, the file is literally called "{timestamp:micros}" as opposed to using the pattern to name the file dynamically, which resulted on lost data every time we tried to sync large tables.
Using {timestamp:millis} works as expected...
Additionally, the default is {date} which leads to the same issue where the file is duplicated leading to data loss. This was reported back in 2023 but never changed.
Timestamp should just be the default.
Relevant log output
Contribute
- Yes, I want to contribute