Skip to content

HDDS-11221. Resolve potential time discrepancy for expired multipart upload cleanup #6982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ivandika3 merged 1 commit into from
Jul 30, 2024
Merged

HDDS-11221. Resolve potential time discrepancy for expired multipart upload cleanup #6982

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions ...p-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
Expand Down Expand Up @@ -1815,7 +1814,7 @@ public List<ExpiredMultipartUploadsBucket> getExpiredMultipartUploads(
mpuInfoTableIterator = getMultipartInfoTable().iterator()) {

final long expiredCreationTimestamp =
Instant.now().minus(expireThreshold).toEpochMilli();
expireThreshold.negated().plusMillis(Time.now()).toMillis();
Copy link
Contributor

@kerneltime kerneltime Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://issues.apache.org/jira/browse/HDDS-7764 lists going to Time.now() as a temporary solution. We should switch to UTC time and not be dependent on local time zone configuration.

Copy link
Contributor Author

@ivandika3 ivandika3 Jul 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review. I agree with switching from Time.now() to Instant.now() (which will be addressed in HDDS-7911). However, currently the issue is that OmKeyInfo creationTimestamp and modificationTimestamp already using Time.now(). If we switch to Instant.now() instead, the timestamps in the key / file table will be a mix of UTC and local time zone.


ExpiredMultipartUploadInfo.Builder builder =
ExpiredMultipartUploadInfo.newBuilder();
Expand Down
5 changes: 2 additions & 3 deletions ...p-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@

import java.io.File;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
Expand Down Expand Up @@ -777,15 +776,15 @@ private void testGetExpiredMPUs() throws Exception {
final Duration expireThreshold = Duration.ofMillis(expireThresholdMillis);

final long expiredMPUCreationTime =
Instant.now().minus(expireThreshold).toEpochMilli();
expireThreshold.negated().plusMillis(Time.now()).toMillis();

// Add expired MPUs to multipartInfoTable.
// The method under test does not check for expired open keys in the
// cache, since they will be picked up once the cache is flushed.
Set<String> expiredMPUs = new HashSet<>();
for (int i = 0; i < numExpiredMPUs + numUnexpiredMPUs; i++) {
final long creationTime = i < numExpiredMPUs ?
expiredMPUCreationTime : Instant.now().toEpochMilli();
expiredMPUCreationTime : Time.now();

String uploadId = OMMultipartUploadUtils.getMultipartUploadId();
final OmMultipartKeyInfo mpuKeyInfo = OMRequestTestUtils
Expand Down