HDDS-426. Add field modificationTime for Volume and Bucket #164
Conversation
There was a problem hiding this comment.
@cxorm Thank you for working on this. Overall LGTM, however, some of the tests don't validate modification time.
Ideally for some of the test like SetQuota, SetOwner etc, you would want to validate that after the operation the modificationTime is > creationTime.
This approach will confirm the modification time is being set appropriately.
Thanks @dineshchitlangia for the comment. |
|
/retest |
|
The UT's error seems not related to the PR. |
dineshchitlangia
left a comment
dineshchitlangia
left a comment
There was a problem hiding this comment.
+1 LGTM. @anuengineer / @arp7 request your review as well.
Thanks @dineshchitlangia for the review. |
|
Hi @anuengineer , |
|
This patch is not complete now. |
|
ok, I will wait for a rebase. Thanks |
cxorm
force-pushed
the
HDDS-426
branch
2 times, most recently
from
ed2803a to
c609f35
Compare
|
+1 LGTM. |
There was a problem hiding this comment.
NIT: Should we use Time.now() which is a wrapper of System.currentTimeMillis consistently across the patch? This will make it easier to maintain the code in the long run. Otherwise, LGTM.
There was a problem hiding this comment.
Thanks @xiaoyuyao for the review.
Replace it with setModificationTime(), and update the corresponding call.
There was a problem hiding this comment.
Modification time should be set in the preExecute, and use the same value in validateAndUpdateCache. why it needs to be done like this is, in HA all OM's should have the same modification time, in this case there is a chance of OM's having different modification time because it depends on the time when validateAndUpdateCache will be executed on OM.
There was a problem hiding this comment.
Thanks @bharatviswa504 for the info.
This info is important for me.
Updated with setting preExecute.
There was a problem hiding this comment.
Modification time should be set in the preExecute, and use the same value in validateAndUpdateCache. why it needs to be done like this is, in HA all OM's should have the same modification time, in this case there is a chance of OM's having different modification time because it depends on the time when validateAndUpdateCache will be executed on OM.
There was a problem hiding this comment.
Thanks @bharatviswa504 for the review.
Updated with setting preExecute.
There was a problem hiding this comment.
Here do we need to get the value once, and set the same value for creatiionTime and ModificationTime. Because once after createBucket, there is a chance of these having different values.
long updateTime = Time.now(), and then use this in setting mofication and creation time.
There was a problem hiding this comment.
Thanks @bharatviswa504 for the comment.
Fixed.
There was a problem hiding this comment.
Here do we need to get the value once, and set the same value for creatiionTime and ModificationTime. Because once after createBucket, there is a chance of these having different values.
long updateTime = Time.now(), and then use this in setting mofication and creation time.
Same as below.
bharatviswa504
left a comment
bharatviswa504
left a comment
There was a problem hiding this comment.
One generic comment modification time should be set in PreExecute and use that in ValidateAndUpdateCache. (Similar to create requests)
|
This PR would be redone and fixed soon. |
|
Some tests would be updated soon. |
|
Description of this PR was updated and execution snapshots were uploaded in JIRA. @bharatviswa504 , @anuengineer Could you help review this PR if you have a time ? |
There was a problem hiding this comment.
Do we consider the backward compatibility here when changing the public client APIs? This will be important moving forward from Beta to GA.
There was a problem hiding this comment.
Thanks @xiaoyuyao for the idea.
The backward compatibility is updated.
xiaoyuyao
left a comment
xiaoyuyao
left a comment
There was a problem hiding this comment.
Thanks @cxorm for the patch. LGTM overall, a few comments added inline. Also, there are other volume/bucket metadata/ACL update APIs that will need similar modification time update. This could be done in a separate JIRA giving the size of the current one. Also, we need to document this modification time only apply to volume/bucket metadata modification itself, not covering key changes under the volume bucket.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
Outdated
Show resolved
Hide resolved
|
Thank you @xiaoyuyao for reviewing this PR. I think the current violation is not related the change,/. |
|
Hi @xiaoyuyao, |
There was a problem hiding this comment.
This can be simplified using Instant without conversions.
this.modifiedTime = Instant.now();
if (modifiedTime.isBefore(this.creationTime)) {
this.modificationTime = this.creationTime.clone();
}
There was a problem hiding this comment.
similar as above to avoid Time.now()
There was a problem hiding this comment.
similar as above to avoid Time.now()
There was a problem hiding this comment.
similar as above to avoid Time.now()
xiaoyuyao
left a comment
xiaoyuyao
left a comment
There was a problem hiding this comment.
LGTM, just few additional minor comments and I think we are ready for the commit. Thanks for the patience for this one.
Codecov Report
@@ Coverage Diff @@
## master #164 +/- ##
============================================
- Coverage 70.56% 70.38% -0.18%
+ Complexity 9427 9419 -8
============================================
Files 965 967 +2
Lines 49063 49352 +289
Branches 4803 4831 +28
============================================
+ Hits 34620 34737 +117
- Misses 12137 12277 +140
- Partials 2306 2338 +32 Continue to review full report at Codecov.
|
|
Thank you @xiaoyuyao for the detailed review. |
|
LGTM, +1. Thanks @cxorm for the patience. I will merge the PR shortly. |
|
Thank you @xiaoyuyao for taking time to review this PR again and again : ) |
8 participants
What changes were proposed in this pull request?
We would update the
modificationTimeofvolumewhenvolume createvolume update (set owner/quota)And we would update the
modificationTimeofbucketwhenbucket createSo we focus on these items.
Proposed fix.
This PR aims to add filed of
modificationTimeto volume and bucket.For volume part, besides adding field in volume, we have 3 requests to be fixed that are
OMVolumeCreateRequest,OMVolumeSetOwnerRequestandOMVolumeSetQuotaRequest.Because
modificationTimeshould be consistent among OM,we refer Handling Write Requests with OM HA to fix these requests. (By setting
modificationTimeinpreExecute(), and using it for late validation.)After setting Owner/Quota of volume, the original
UpdateVolumeHandlerwould print oldmodificationTimecause its volume is not updated volume(just updated owner/quota).We should get volume again for getting newer
modification.For bucket part, we should fix
OMBucketCreateRequest.And for consistent,
we set
modificationTimeof creating bucket inOMBucketCreateRequest#preExecute().What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-426
How was this patch tested?
modificationTimefield snapshots uploaded in JIRA.