Skip to content

HADOOP-16914 Adding Output Stream Counters in ABFS #1899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
steveloughran merged 5 commits into from
Apr 23, 2020
Merged

HADOOP-16914 Adding Output Stream Counters in ABFS #1899

steveloughran merged 5 commits into from
Apr 23, 2020

Conversation

@mehakmeet
Copy link
Contributor

@mehakmeet mehakmeet commented Mar 17, 2020

Change-Id: Ie976f23c6c3e2f5cf9167794357cfc669a232c80

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

@mehakmeet
Copy link
Contributor Author

mehakmeet commented Mar 17, 2020 •
edited
Loading

test run : mvn -T 1C -Dparallel-tests=abfs clean verify
Region : East US, West US

Failed tests(Container configurations based) :

[ERROR] Failures: [ERROR] ITestGetNameSpaceEnabled.testNonXNSAccount:59->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:88 Expecting getIsNamespaceEnabled() return false
[ERROR] Errors: [ERROR] ITestGetNameSpaceEnabled.testFailedRequestWhenCredentialsNotCorrect:91->AbstractAbfsIntegrationTest.getFileSystem:197 ? InvalidConfigurationValue

@mehakmeet
Copy link
Contributor Author

mehakmeet commented Mar 17, 2020

Please suggest how to write tests for "timeSpendOnTaskWait" counter.
Please suggest how to simulate bytes failed to upload for "bytesUploadFailed" counter to be populated.
flush() is an expensive call but need to call it to see counters to populate, hence only 1000 operation is called "Large"(takes 1-3 minutes per test if 1000 flush() are called). Should I leave it as it is, or is there another way ?

bgaborg
bgaborg requested changes Mar 17, 2020
View reviewed changes
Copy link

@bgaborg bgaborg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @mehakmeet!

Compile is failing for me locally. Are you sure that you were able to run the test with this?
There are also 21 new checkstyle errors introduced, be sure that you fix those as well.

@mehakmeet mehakmeet requested a review from bgaborg March 18, 2020 09:19
steveloughran
steveloughran requested changes Mar 20, 2020
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

coming along; made comments on production and test code.

regarding the time to run the tests -can you just run a few times rather than thousands of times?

@mehakmeet
Copy link
Contributor Author

mehakmeet commented Apr 1, 2020 •
edited
Loading

getting timeout errors in abfs-tests(might be server issue):

[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemDelete.testDeleteFirstLevelDirectory:127->lambda$testDeleteFirstLevelDirectory$1:128 » TestTimedOut
[ERROR]   ITestAzureBlobFileSystemMainOperation>FSMainOperationsBaseTest.testDeleteEmptyDirectory:787 » AbfsRestOperation
[ERROR]   ITestAzureBlobFileSystemMainOperation>FSMainOperationsBaseTest.testRenameDirectoryAsEmptyDirectory:1033->FSMainOperationsBaseTest.rename:1154 » AbfsRestOperation


[ERROR] Errors:
[ERROR]   ITestAbfsFileSystemContractDelete>AbstractContractDeleteTest.testDeleteEmptyDirNonRecursive:37->AbstractFSContractTestBase.assertDeleted:369 » TestTimedOut
[ERROR]   ITestAbfsFileSystemContractDelete>AbstractContractDeleteTest.testDeleteNonEmptyDirNonRecursive:77 » TestTimedOut
[ERROR]   ITestAbfsFileSystemContractMkdir>AbstractContractMkdirTest.testMkDirRmDir:46->AbstractFSContractTestBase.assertDeleted:369 » TestTimedOut
[ERROR]   ITestAzureBlobFileSystemBasics>FileSystemContractBaseTest.testDeleteEmptyDirectory:441 » TestTimedOut
[ERROR]   ITestAzureBlobFileSystemBasics>FileSystemContractBaseTest.testDeleteRecursively:421 » TestTimedOut
[ERROR]   ITestAzureBlobFileSystemBasics>FileSystemContractBaseTest.testFilesystemIsCaseSensitive:651 » TestTimedOut
[ERROR]   ITestAzureBlobFileSystemBasics.setUp:49 » TestTimedOut test timed out after 30...
[ERROR]   ITestAzureBlobFileSystemBasics>FileSystemContractBaseTest.testRenameDirectoryAsExistingDirectory:548->FileSystemContractBaseTest.createFile:590 » TestTimedOut
[ERROR]   ITestAzureBlobFileSystemBasics>FileSystemContractBaseTest.testRenameDirectoryMoveToExistingDirectory:515->FileSystemContractBaseTest.createFile:590 » TestTimedOut
[ERROR]   ITestAzureBlobFileSystemBasics.tearDown:61 » TestTimedOut test timed out after...

mukund-thakur
mukund-thakur requested changes Apr 2, 2020
View reviewed changes
Copy link
Contributor

@mukund-thakur mukund-thakur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just improve the java doc. Will review again once done. Thanks

@mehakmeet mehakmeet force-pushed the HADOOP-16914 branch 2 times, most recently from c8c6608 to 2bada7f Compare April 6, 2020 12:03
@mehakmeet mehakmeet requested a review from steveloughran April 6, 2020 13:08
mukund-thakur
mukund-thakur reviewed Apr 7, 2020
View reviewed changes
...op-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java Outdated
Copy link
Contributor

@mukund-thakur mukund-thakur Apr 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this an UT. Should move under UT folder. Create a new class TestAbfsOutputStreamStatictics and write all UT's there. You don't even have to create a file there. What do you say @steveloughran?

@apache apache deleted a comment from hadoop-yetus Apr 20, 2020
@apache apache deleted a comment from hadoop-yetus Apr 20, 2020
@apache apache deleted a comment from hadoop-yetus Apr 20, 2020
steveloughran
steveloughran requested changes Apr 20, 2020
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is almost ready to go in. My one change is that we don't expose the statistics Impl class from the production-side code; the test code will have to do the casting, which it can do via some static method if that simplifies life

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java Outdated
Copy link
Contributor

@steveloughran steveloughran Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this would be coding into the implementation/semi-public API something only relevant for testing -and make it very hard to ever change to a different implementation.

Just add some static method in the test suite

AbfsOutputStreamStatisticsImp getStreamStatistics(AbfsOutputStream)

and you could do the casting in just one place.

@mehakmeet mehakmeet requested a review from steveloughran April 21, 2020 07:57
@hadoop-yetus
Copy link

hadoop-yetus commented Apr 21, 2020

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 30m 4s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 22m 10s trunk passed
+1 💚 compile 0m 28s trunk passed
+1 💚 checkstyle 0m 23s trunk passed
+1 💚 mvnsite 0m 33s trunk passed
+1 💚 shadedclient 16m 11s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 23s trunk passed
+0 🆗 spotbugs 0m 53s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 50s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 26s the patch passed
+1 💚 compile 0m 22s the patch passed
+1 💚 javac 0m 22s the patch passed
+1 💚 checkstyle 0m 14s the patch passed
+1 💚 mvnsite 0m 25s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 35s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 20s the patch passed
+1 💚 findbugs 0m 55s the patch passed
_ Other Tests _
+1 💚 unit 1m 18s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 27s The patch does not generate ASF License warnings.
92m 30s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/12/artifact/out/Dockerfile
GITHUB PR #1899
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 2743c444e985 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 60fa153
Default Java Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/12/testReport/
Max. process+thread count 325 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/12/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Apr 22, 2020

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 7m 23s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 24m 14s trunk passed
+1 💚 compile 0m 29s trunk passed
+1 💚 checkstyle 0m 21s trunk passed
+1 💚 mvnsite 0m 31s trunk passed
+1 💚 shadedclient 16m 40s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 23s trunk passed
+0 🆗 spotbugs 0m 51s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 49s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 27s the patch passed
+1 💚 compile 0m 23s the patch passed
+1 💚 javac 0m 23s the patch passed
+1 💚 checkstyle 0m 15s the patch passed
+1 💚 mvnsite 0m 26s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 29s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 22s the patch passed
+1 💚 findbugs 0m 55s the patch passed
_ Other Tests _
+1 💚 unit 1m 17s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
72m 32s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/13/artifact/out/Dockerfile
GITHUB PR #1899
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux b27916ec7a55 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3d69383
Default Java Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/13/testReport/
Max. process+thread count 299 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/13/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Apr 22, 2020

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 9s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 21m 37s trunk passed
+1 💚 compile 0m 28s trunk passed
+1 💚 checkstyle 0m 20s trunk passed
+1 💚 mvnsite 0m 30s trunk passed
+1 💚 shadedclient 16m 14s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 22s trunk passed
+0 🆗 spotbugs 0m 48s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 47s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 27s the patch passed
+1 💚 compile 0m 22s the patch passed
+1 💚 javac 0m 22s the patch passed
+1 💚 checkstyle 0m 15s the patch passed
+1 💚 mvnsite 0m 25s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 31s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 20s the patch passed
+1 💚 findbugs 0m 54s the patch passed
_ Other Tests _
+1 💚 unit 1m 19s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 28s The patch does not generate ASF License warnings.
62m 58s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/14/artifact/out/Dockerfile
GITHUB PR #1899
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 71cef5d39d33 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3d69383
Default Java Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/14/testReport/
Max. process+thread count 346 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/14/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Apr 23, 2020
steveloughran
steveloughran requested changes Apr 23, 2020
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all good apart from that input

@mehakmeet
Copy link
Contributor Author

mehakmeet commented Apr 23, 2020

Sorry for the force push, I had done the -amend and rebased it afterwards. So, wasn't able to go back to the previous HEAD to have the commit and unstage changes.

@mehakmeet mehakmeet requested a review from steveloughran April 23, 2020 11:51
@steveloughran
Copy link
Contributor

steveloughran commented Apr 23, 2020 •
edited
Loading

Sorry for the force push, I had done the -amend and rebased it afterwards. So, wasn't able to go back to the previous HEAD to have the commit and unstage changes.

why the -amend? Why not just add another cvhange.

Please don't rebase once reviewing has started, as it becomes impossible to tie discussions back to the state of the patch at the time, or see what changes happened after. For example, a commit called "fix review changes" Which review? I can't see from the history any more

Until other people start reviewing -go for it. Once it's begun, if you do need to reset everything it is better to start again with a whole new PR with the single history squashed

git diff trunk...HEAD > history.diff
git co trunk
git co -b new-branch
git apply -3 --verbose --whitespace=fix history.diff
..etc. etc

Key point; rebasing makes reviewing significantly harder, the harder a patch is to review, the fewer reviews it gets.

@steveloughran steveloughran merged commit 459eb2a into apache:trunk Apr 23, 2020
@hadoop-yetus
Copy link

hadoop-yetus commented Apr 23, 2020

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 19m 18s trunk passed
+1 💚 compile 0m 31s trunk passed
+1 💚 checkstyle 0m 24s trunk passed
+1 💚 mvnsite 0m 35s trunk passed
+1 💚 shadedclient 14m 51s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 26s trunk passed
+0 🆗 spotbugs 0m 51s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 49s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 23s the patch passed
+1 💚 javac 0m 23s the patch passed
+1 💚 checkstyle 0m 17s the patch passed
+1 💚 mvnsite 0m 27s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 2s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 23s the patch passed
+1 💚 findbugs 0m 53s the patch passed
_ Other Tests _
+1 💚 unit 1m 23s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 31s The patch does not generate ASF License warnings.
57m 39s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/15/artifact/out/Dockerfile
GITHUB PR #1899
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 10e816d6c9d8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 5958af4
Default Java Private Build-1.8.0_252-8u252-b09-1~16.04-b09
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/15/testReport/
Max. process+thread count 422 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/15/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor

steveloughran commented Apr 23, 2020

+1, merged to trunk

please don't do rebase/amend/force push again. thanks

asfgit pushed a commit that referenced this pull request Apr 23, 2020
@mehakmeet @steveloughran
HADOOP-16914 Adding Output Stream Counters in ABFS (#1899)
2471ba8 
Contributed by Mehakmeet Singh.There
arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request Apr 12, 2022
@mehakmeet @arjun4084346
HADOOP-16914 Adding Output Stream Counters in ABFS (apache#1899)
f06be90 
Contributed by Mehakmeet Singh.There

(cherry picked from commit 459eb2a)
arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request Apr 15, 2022
@mehakmeet @arjun4084346
HADOOP-16914 Adding Output Stream Counters in ABFS (apache#1899)
3a22cc6 
Contributed by Mehakmeet Singh.There

(cherry picked from commit 459eb2a)
arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request May 3, 2022
@mehakmeet @arjun4084346
HADOOP-16914 Adding Output Stream Counters in ABFS (apache#1899)
957e75d 
Contributed by Mehakmeet Singh.There

(cherry picked from commit 459eb2a)
zhangxiping1 pushed a commit to zhangxiping1/hadoop that referenced this pull request Dec 13, 2022
@mehakmeet @zhangxiping1
HADOOP-16914 Adding Output Stream Counters in ABFS (apache#1899)
e9672cd 
Contributed by Mehakmeet Singh.There
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mukund-thakur mukund-thakur mukund-thakur requested changes

@bgaborg bgaborg Awaiting requested review from bgaborg

@steveloughran steveloughran Awaiting requested review from steveloughran

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mehakmeet @hadoop-yetus @steveloughran @bgaborg @mukund-thakur