HDDS-4514. AllocateBlock : lookup and update open file table for the given path #1679

rakeshadr · 2020-12-09T12:20:34Z

What changes were proposed in this pull request?

This task is to use open file table for the allocate block operations. This has been identified as part of Hive TPCDS benchmark test

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4514

How was this patch tested?

Added UT cases. Also, verified by running hive tpcds test

…given path

linyiqun

Please address below minor comment and fix failed unit test, : ).

...manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponseV1.java

…ck UT failure

rakeshadr · 2020-12-09T17:40:16Z

Please address below minor comment and fix failed unit test, : ).

TestOzoneFileSystemV1#testListStatusOnLargeDirectory is failing in the build, interestingly it is passing consistently in my local env. Added log messages to debug it. Will monitor the results in next build.

…ctory to fix failure

rakeshadr · 2020-12-10T14:40:31Z

@linyiqun I've addressed your comments. Could you please review it again when you get a chance. Thanks!

linyiqun

+1.
Thanks for addressing the comments, @rakeshadr .

...e-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequestV1.java

bharatviswa504

Overall LGTM, few comments.

...ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java

bharatviswa504 · 2020-12-11T18:17:45Z

...ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java

+  private static void checkDirectoryAlreadyExists(String keyName,
+      OzoneManager ozoneManager, boolean reachedLastPathComponent)
+          throws IOException {
+    // Reached last component, which would be a file. Returns its parentID.


Minor: Comment says Returns its parentID.

In code we just check if lastComponent is true and fileSystem enabled, we return exception.

And also as current V1 always assumes fs enabled true, do we need && ozoneManager.getEnableFileSystemPaths() check?

And also these 3 lines can be in the calling method, as it is not a utility method which is called by different classes.

Agreed, Done the changes!

bharatviswa504 · 2020-12-11T18:35:26Z

...one-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java

-      omClientResponse = new OMAllocateBlockResponse(omResponse.build(),
-          openKeyInfo, clientID, omVolumeArgs, omBucketInfo.copyObject());

+      omClientResponse = getOmClientResponse(clientID, omResponse,


We don't update omVolumeArgs, but still reading and passing to Response, which is not required.
Not related to your patch BTW, we can fix this in master and get to branch.

...ger/src/test/java/org/apache/hadoop/ozone/om/response/key/TestOMAllocateBlockResponseV1.java

bharatviswa504

+1 LGTM

...one-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java

rakeshadr · 2020-12-22T03:31:47Z

Thank you @linyiqun , @bharatviswa504 for the useful review comments. Merging it to the feature branch.

…given path (#1679)

HDDS-4514. AllocateBlock : lookup and update open file table for the …

2eba8a7

…given path

rakeshadr mentioned this pull request Dec 9, 2020

HDDS-4514. AllocateBlock : lookup and update open file table for the given path #1656

Closed

Updated RequestV1 test case to use new prefix tables

8ab5ea1

linyiqun reviewed Dec 9, 2020

View reviewed changes

...manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponseV1.java Outdated Show resolved Hide resolved

Fixed review comment - removed volume usedbytes and added logs to che…

dc40da0

…ck UT failure

rakeshadr requested a review from bharatviswa504 December 10, 2020 02:17

rakeshadr added 2 commits December 10, 2020 08:27

Added rootDirCleanup on TestOzoneFileSystem#testListStatusOnLargeDire…

37d2039

…ctory to fix failure

Fixed checkstyle warning

d6130b4

linyiqun approved these changes Dec 10, 2020

View reviewed changes

bharatviswa504 reviewed Dec 11, 2020

View reviewed changes

...e-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequestV1.java Show resolved Hide resolved

bharatviswa504 reviewed Dec 11, 2020

View reviewed changes

...e-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequestV1.java Show resolved Hide resolved

bharatviswa504 reviewed Dec 11, 2020

View reviewed changes

rakeshadr added 4 commits December 21, 2020 11:07

Fixed Bharat's review comments

4b0c26b

Fixed compilation error

3c46fb1

Fixed checkstyle warnings

702e038

Added omBucketInfo.copyObject() in OMAllocateBlockRequest

82cec8d

bharatviswa504 reviewed Dec 21, 2020

View reviewed changes

...one-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java Show resolved Hide resolved

Removed unnecessary spaces in OMAllocateBlockRequest

aaf77f4

rakeshadr merged commit 9618c5f into apache:HDDS-2939 Dec 22, 2020

rakeshadr added a commit that referenced this pull request Jan 6, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

48c925a

…given path (#1679)

rakeshadr added a commit that referenced this pull request Jan 11, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

bd69d47

…given path (#1679)

rakeshadr added a commit that referenced this pull request Jan 13, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

62d01b8

…given path (#1679)

rakeshadr added a commit that referenced this pull request Jan 25, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

09023c4

…given path (#1679)

rakeshadr added a commit that referenced this pull request Jan 28, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

692c010

…given path (#1679)

rakeshadr added a commit that referenced this pull request Feb 1, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

5724590

…given path (#1679)

rakeshadr added a commit that referenced this pull request Feb 1, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

e32813f

…given path (#1679)

rakeshadr added a commit that referenced this pull request Feb 3, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

d5a930d

…given path (#1679)

rakeshadr added a commit that referenced this pull request Feb 5, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

f39a707

…given path (#1679)

rakeshadr added a commit that referenced this pull request Feb 15, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

5cffb69

…given path (#1679)

rakeshadr added a commit that referenced this pull request Feb 18, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

fd94a0e

…given path (#1679)

rakeshadr added a commit that referenced this pull request Feb 25, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

90e8362

…given path (#1679)

rakeshadr added a commit that referenced this pull request Mar 3, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

3045ec6

…given path (#1679)

rakeshadr added a commit that referenced this pull request Mar 9, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

4248adb

…given path (#1679)

rakeshadr added a commit that referenced this pull request Mar 27, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

be61797

…given path (#1679)

rakeshadr added a commit that referenced this pull request Apr 7, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

aec4ca5

…given path (#1679)

rakeshadr added a commit that referenced this pull request Apr 8, 2021

HDDS-4514. AllocateBlock : lookup and update open file table for the …

c10abe6

…given path (#1679)

HDDS-4514. AllocateBlock : lookup and update open file table for the given path #1679

HDDS-4514. AllocateBlock : lookup and update open file table for the given path #1679

Uh oh!

Conversation

rakeshadr commented Dec 9, 2020

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

linyiqun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rakeshadr commented Dec 9, 2020

Uh oh!

rakeshadr commented Dec 10, 2020

Uh oh!

linyiqun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bharatviswa504 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bharatviswa504 Dec 11, 2020

Choose a reason for hiding this comment

Uh oh!

rakeshadr Dec 21, 2020

Choose a reason for hiding this comment

Uh oh!

bharatviswa504 Dec 11, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bharatviswa504 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rakeshadr commented Dec 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

linyiqun left a comment •

edited

Loading